21 Hottest Research Papers On Computer Vision and Machine Learning These papers provide a breadth of information about powerful convolutional neural networks that is generally useful and interesting from a Data science perspective. Contents 1. HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition 2. Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture 3. High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision 4. Dense Optical Flow Prediction from a Static Image 5. Ask Your Neurons: A Neural-based Approach to Answering Questions about Images 6. Learning to See by Moving 7. Unsupervised Visual Representation Learning by Context Prediction 8. PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization 9. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books 10. Deep Networks for Image Super-Resolution with Sparse Prior 11. A Deep Visual Correspondence Embedding Model for Stereo Matching Costs 12. Conditional Random Fields as Recurrent Neural Networks 13. Local Convolutional Features with Unsupervised Training for Image Retrieval 14. FlowNet: Learning Optical Flow with Convolutional Networks 15. Active Object Localization with Deep Reinforcement Learning 16. Deep Neural Decision Forests 17. Im2Calories: towards an automated mobile vision food diary 18. DeepBox: Learning Objectness with Convolutional Networks 19. Flowing ConvNets for Human Pose Estimation in Videos 20. Understanding deep features with computer-generated imagery 21. Visual Tracking with Fully Convolutional Networks HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition arXiv:1410.0736v4 [cs.CV] 16 May 2015 Zhicheng Yan† , Hao Zhang‡ , Robinson Piramuthu∗ , Vignesh Jagadeesh∗ , Dennis DeCoste∗ , Wei Di∗ , Yizhou Yu? † University of Illinois at Urbana-Champaign, ‡ Carnegie Mellon University ∗ eBay Research Lab, ? The University of Hong Kong Abstract harder to tell an Apple from an Orange. In fact, both Apples and Oranges belong to the same coarse category fruit and vegetables while Buses belong to another coarse category vehicles 1, as defined within CIFAR100. Nonetheless, most deep CNN models nowadays are flat N-way classifiers, which share a set of fully connected layers. This makes us wonder whether such a flat structure is adequate for distinguishing all the difficult categories. A very natural and intuitive alternative organizes classifiers in a hierarchical manner according to the divide-and-conquer strategy. Although hierarchical classification has been proven effective for conventional linear classifiers [38, 8, 37, 22], few attempts have been made to exploit category hierarchies [3, 29] in deep CNN models. Since deep CNN models are large models themselves, organizing them hierarchically imposes the following challenges. First, instead of a handcrafted category hierarchy, how can we learn such a category hierarchy from the training data itself so that cascaded inferences in a hierarchical classifier will not degrade the overall accuracy while dedicated fine category classifiers exist for hard-to-distinguish categories? Second, a hierarchical CNN classifier consists of multiple CNN models at different levels. How can we leverage the commonalities among these models and effectively train them all? Third, it would also be slower and more memory-consuming to run a hierarchical CNN classifier on a novel testing image. How can we alleviate such limitations? In this paper, we propose a generic and principled hierarchical architecture, Hierarchical Deep Convolutional Neural Network (HD-CNN), that decomposes an image classification task into two steps. An HD-CNN first uses a coarse category CNN classifier to separate easy classes from one another. More challenging classes are routed downstream to fine category classifiers that focus on confusing classes. We adopt a module design principle and every HD-CNN is built upon a building block CNN. The building block can be chosen to be any of the currently top ranked single CNNs. Thus HD-CNNs can always benefit from the progress of single CNN design. An HD-CNN follows the coarse-to-fine In image classification, visual separability between different object categories is highly uneven, and some categories are more difficult to distinguish than others. Such difficult categories demand more dedicated classifiers. However, existing deep convolutional neural networks (CNN) are trained as flat N-way classifiers, and few efforts have been made to leverage the hierarchical structure of categories. In this paper, we introduce hierarchical deep CNNs (HD-CNNs) by embedding deep CNNs into a category hierarchy. An HD-CNN separates easy classes using a coarse category classifier while distinguishing difficult classes using fine category classifiers. During HD-CNN training, component-wise pretraining is followed by global finetuning with a multinomial logistic loss regularized by a coarse category consistency term. In addition, conditional executions of fine category classifiers and layer parameter compression make HD-CNNs scalable for large-scale visual recognition. We achieve state-of-the-art results on both CIFAR100 and large-scale ImageNet 1000-class benchmark datasets. In our experiments, we build up three different HD-CNNs and they lower the top-1 error of the standard CNNs by 2.65%, 3.1% and 1.1%, respectively. 1. Introduction Deep CNNs are well suited for large-scale learning based visual recognition tasks because of its highly scalable training algorithm, which only needs to cache a small chunk (mini-batch) of the potentially huge volume of training data during sequential scans (epochs). They have achieved increasingly better performance in recent years. As datasets become bigger and the number of object categories becomes larger, one of the complications that come along is that visual separability between different object categories is highly uneven. Some categories are much harder to distinguish than others. Take the categories in CIFAR100 as an example. It is easy to tell an Apple from a Bus, but 4321 Root Coarse component independent layers B Fine component independent layers F1 Coarse category 2: {toy terrier, greyhound, whippet, basenji, … } .   .   .   Coarse category K: {mink, cougar, bear, fox squirrel, otter, … } Image Shared layers .   .   .   low-level features Fine component independent layers FK (a) fine prediction 1 fine prediction K Probabilistic averaging layer coarse prediction Coarse category 1: {white shark, numbfish, hammerhead, stingray, … } final prediction (b) Figure 1: (a) A two-level category hierarchy where the classes are taken from ImageNet 1000-class dataset. (b) Hierarchical Deep Convolutional Neural Network (HD-CNN) architecture. 2.2. Category Hierarchy for Visual Recognition classification paradigm and probabilistically integrates predictions from the fine category classifiers. Compared with the building block CNN, the corresponding HD-CNN can achieve lower error at the cost of a manageable increase in memory footprint and classification time. In summary, this paper has the following contributions. First, we introduce a novel hierarchical architecture, called HD-CNN, for image classification. Second, we develop a scheme for learning the two-level organization of coarse and fine categories, and demonstrate various components of an HD-CNN can be independently pretrained. The complete HD-CNN is further fine-tuned using a multinomial logistic loss regularized by a coarse category consistency term. Third, we make the HD-CNN scalable by compressing the layer parameters and conditionally executing the fine category classifiers. We have performed evaluations on the medium-scale CIFAR100 dataset and the large-scale ImageNet 1000-class dataset, and our method has achieved state-of-the-art performance on both of them. In visual recognition, there is a vast literature exploiting category hierarchical structures [32]. For classification with a large number of classes using linear classifiers, a common strategy is to build a hierarchy or taxonomy of classifiers so that the number of classifiers evaluated given a testing image scales sub-linearly in the number of classes [2, 9]. The hierarchy can be either predefined[23, 33, 16] or learnt by top-down and bottom-up approaches [25, 12, 24, 20, 1, 6, 27]. In [5], the predefined category hierarchy of ImageNet dataset is utilized to achieve the trade-offs between classification accuracy and specificity. In [22], a hierarchical label tree is constructed to probabilistically combine predictions from leaf nodes. Such hierarchical classifier achieves significant speedup at the cost of certain accuracy loss. One of the earliest attempts to introduce a category hierarchy in CNN-based methods is reported in [29] but their main goal is transferring knowledge between classes to improve the results for classes with insufficient training examples. In [3], various label relations are encoded in a hierarchy. Improved accuracy is achieved only when a subset of training images are relabeled with internal nodes in the hierarchical class tree. They are not able to improve the accuracy in the original setting where all training images are labeled with leaf nodes. In [34], a hierarchy of CNNs is introduced but they experimented with only two coarse categories mainly due to scalability constraints. HDCNN exploits the category hierarchy in a novel way that we embed deep CNNs into the hierarchy in a scalable manner and achieves superior classification results over the standard CNN. 2. Related Work Our work is inspired by progresses in CNN design and efforts on integrating a category hierarchy with linear classifiers. The main novelty of our method is a new scalable HD-CNN architecture that integrates a category hierarchy with deep CNNs. 2.1. Convolutional Neural Networks CNN-based models hold state-of-the-art performance in various computer vision tasks, including image classifcation [18], object detection [10, 13], and image parsing [7]. Recently, there has been considerable interest in enhancing CNN components, including pooling layers [36], activation units [11, 28], and nonlinear layers [21]. These enhancements either improve CNN training [36], or expand the network learning capacity. This work boosts CNN performance from an orthogonal angle and does not redesign a specific part within any existing CNN model. Instead, we design a novel generic hierarchical architecture that uses an existing CNN model as a building block. We embed multiple building blocks into a larger hierarchical deep CNN. 3. Overview of HD-CNN 3.1. Notations The following notations are used below. A dataset consists of images {xi , yi }i . xi and yi denote the image data and label, respectively. There are C fine categories of images in the dataset {Sjf }C j=1 . We will learn a category hierarchy with K coarse categories {Skc }K k=1 . 4322 3.2. HD-CNN Architecture as the final prediction. PK HD-CNN is designed to mimic the structure of category hierarchy where fine categories are divided into coarse categories as in Fig 1(a). It performs end-to-end classification as illustrated in Fig 1(b). It mainly comprises four parts, namely shared layers, a single coarse category component B, multiple fine category components {F k }K k=1 and a single probabilistic averaging layer. On the left side of Fig 1 (b) are the shared layers. They receive raw image pixel as input and extract low-level features. The configuration of shared layers are set to be the same as the preceding layers in the building block net. On the top of Fig 1(b) are independent layers of coarse category component B which reuses the configuration of rear layers from the building block CNN and produces a fine f C }j=1 for an image xi . To produce a predicprediction {Bij K tion {Bik }k=1 over coarse categories, we append a fine-tocoarse aggregation layer which aggregates fine predictions into coarse ones when a mapping from fine categories to coarse ones P : [1, C] 7→ [1, K] is given. The coarse category probabilities serve two purposes. First, they are used as weights for combining the predictions made by fine category components. Second, when thresholded, they enable conditional executions of fine category components whose corresponding coarse probabilities are sufficiently large. In the bottom-right of Fig 1 (b) are independent layers of a set of fine category classifiers {F k }K k=1 , each of which makes fine category predictions. As each fine component only excels in classifying a small set of categories, they produce a fine prediction over a partial set of categories. The probabilities of other fine categories absent in the partial set are implicitly set to zero. The layer configurations are mostly copied from the building block CNN except that in the final classification layer the number of filters is set to be the size of partial set instead of the full categories. Both coarse category component and fine category components share common layers. The reason is three-fold. First, it is shown in [35] that preceding layers in deep networks response to class-agnostic low-level features such as corners and edges, while rear layers extract more classspecific features such as dog face and bird’s legs. Since low-level features are useful for both coarse and fine classification tasks, we allow the preceding layers to be shared by both coarse and fine components. Second, it reduces both the total floating point operations and the memory footprint of network execution. Both are of practical significance to deploy HD-CNN in real applications. Last but not the least, it can decrease the number of HD-CNN parameters which is critical to the success of HD-CNN training. On the right side of Fig 1 (b) is the probabilistic averaging layer which receives fine category predictions as well as coarse category prediction and produces a weighted average p(xi ) = k=1 Bik pk (xi ) PK k=1 Bik (1) where Bik is the probability of coarse category k for the image xi predicted by the coarse category component B. pk (xi ) is the fine category prediction made by the fine category component F k . We stress that both coarse and fine category components reuse the layer configurations from the building block CNN. This flexible modular design allows us to choose the best module CNN as the building block, depending on the task at hand. 4. Learning a Category Hierarchy Our goal of building a category hierarchy is grouping confusing fine categories into the same coarse category for which a dedicated fine category classifier will be trained. We employ a top-down approach to learn the hierarchy from the training data. We randomly sample a held-out set of images with balanced class distribution from the training set. The rest of the training set is used to train a building block net. We obtain a confusion matrix F by evaluating the net on the held-out set. A distance matrix D is derived as D = 1 − F and its diagonal entries are set to be zero. D is further transformed by D = 0.5 ∗ (D + DT ) to be symmetric. The entry Dij measures how easy it is to discriminate categories i and j. Spectral clustering is performed on D to cluster fine categories into K coarse categories. The result is a twolevel category hierarchy representing a many-to-one mapping P d : [1, C] 7→ [1, K] from fine to coarse categories. Here, the coarse categories are disjoint. Overlapping Coarse Categories With disjoint coarse categories, the overall classification depends heavily on the coarse category classifier. If an image is routed to an incorrect fine category classifier, then the mistake can not be corrected as the probability of ground truth label is implicitly set to zero there. Removing the separability constraint between coarse categories can make the HD-CNN less dependent on the coarse category classifier. Therefore, we add more fine categories to the coarse categories. For a certain fine classifier F k , we prefer to add those fine categories {j} that are likely to be misclassfied into the coarse category k. Therefore, we estimate the likelihood uk (j) that an image in fine category j is misclassified into a coarse category k on the held-out set. 1 X d uk (j) = Bik (2) Sjf i∈S f j d Bik is the coarse category probability which is obtained by f aggregating fine category probabilities {Bij }j according to 4323 P f d the mapping P d : Bik = j|P d (j)=k Bij . We threshold the k likelihood u (j) using a parametric variable ut = (γK)−1 and add to the partial set Skc all fine categories {j} such that uk (j) ≥ ut . Note that each branching component gives a full set prediction when ut = 0 and a disjoint set prediction when ut = 1.0. With overlapping coarse categories, the category hierarchy mapping P d is extended to be a many-tomany mapping P o and the coarse category predictions are P o updated accordingly Bik = j|k∈P o (j) Bij . Note the sum o K of {Bik }k=1 exceeds 1 and hence we perform L1 normalization. The use of overlapping coarse categories was also shown to be useful for hierarchical linear classifiers [24]. Algorithm 1 HD-CNN training algorithm procedure HD-CNN T RAINING Step 1: Pretrain HD-CNN 3: Step 1.1: Initialize coarse category component 4: Step 1.2: Pretrain fine category components 5: Step 2: Fine-tune the complete HD-CNN 1: 2: each F k , we initialize all the rear layers except the last convolutional layer by copying the learned parameters from the pretrained model F p . 5.2. Fine-tuning HD-CNN After both coarse category component and fine category components are properly pretrained, we fine-tune the complete HD-CNN. Once the category hierarchy as well as the associated mapping P o is learnt, each fine category component focuses on classifying a fixed subset of fine categories. During fine-tuning, the semantics of coarse categories predicted by the coarse category component should be kept consistent with those associated with fine category components. Thus we add a coarse category consistency term to regularize the conventional multinomial logistic loss. Coarse category consistency The learnt fine-to-coarse category mapping P : [1, C] 7→ [1, K] provides a way to specify the target coarse category distribution {tk }. Specifically, tk is set to be the fraction of all the training images within the coarse category Skc under the assumption the distribution over coarse categories across the training dataset is close to that within a training mini-batch. 5. HD-CNN Training As we embed fine category components into HD-CNN, the number of parameters in rear layers grows linearly in the number of coarse categories. Given the same amount of training data, this increases the training complexity and the risk of over-fitting. On the other hand, the training images within the stochastic gradient descent mini-batch are probabilistically routed to different fine category components. It requires larger mini-batch to ensure parameter gradients in the fine category components are estimated by a sufficiently large number of training samples. Large training mini-batch both increases the training memory footprint and slows down the training process. Therefore, we decompose the HD-CNN training into multiple steps instead of training the complete HD-CNN from scratch as outlined in Algorithm 1. P 5.1. Pretraining HD-CNN tk = PK k0 =1 We sequentially pretrain the coarse category component and fine category components. 5.1.1 P |Sj | j|k0 ∈P (j) |Sj | ∀k ∈ [1, K] (3) The final loss function we use for fine-tuning the HDCNN is shown below. Initializing the Coarse Category Component n E=− p We first pretrain a building block CNN F using the training set. As both the preceding and rear layers in coarse category component resemble the layers in the building block CNN, we copy the weights of F p into coarse category component for initialization purpose. 5.1.2 j|k∈P (j) K n 1X λX 1X log(pyi ) + (tk − Bik )2 n i=1 2 n i=1 (4) k=1 where n is the size of training mini-batch. λ is a regularization constant and is set to λ = 20. 6. HD-CNN Testing As we add fine category components into the HD-CNN, the number of parameters, memory footprint and execution time in rear layers, all scale linearly in the number of coarse categories. To ensure HD-CNN is scalable for large-scale visual recognition, we develop conditional execution and layer parameter compression techniques. Conditional Execution. At test time, for a given image, it is not necessary to evaluate all fine category classifiers as most of them have insignificant weights Bik as in Eqn 1. Pretraining the Rear Layers of Fine Category Components Fine category components {F k }k can be independently pretrained in parallel. Each F k should specialize in classifying fine categories within the coarse category Skc . Therefore, the pretraining of each F k only uses images {xi |i ∈ Skc } from the coarse category Skc . The shared preceding layers are already initialized and kept fixed in this stage. For 4324 Their contributions to the final prediction are negligible. Conditional executions of the top weighted fine components can accelerate the HD-CNN classification. Therefore, we threshold Bik using a parametric variable Bt = (βK)−1 and reset Bik to zero when Bik < Bt . Those fine category classifiers with Bik = 0 are not evaluated. Parameter Compression. In HD-CNN, the number of parameters in rear layers of fine category classifiers grows linearly in the number of coarse categories. Thus we compress the layer parameters at test time to reduce memory footprint. Specifically, we choose the Product Quantization approach [14] to compress the parameter matrix W ∈ Rm×n by first partitioning it horizontally into segments of width s. W = [W 1 , ..., W (n/s) ]. Then k-means clustering is used to cluster the rows in W i , ∀i ∈ [1, (n/s)]. By only storing the nearest cluster indices in a 8-bit integer matrix I ∈ Rm×(n/s) and cluster centers in a single-precision floating number matrix C ∈ Rk×n , we can achieve a compression factor (32mn)/(32kn + 8mn/s). The hyperparameters for parameter compression are (s, k). Table 1: 10-view testing errors on CIFAR100 dataset. Notation CCC=coarse category consistency. Error Method Model averaging (2 CIFAR100-NIN nets) DSN [19] CIFAR100-NIN-double dasNet [30] 35.13 34.68 34.26 33.78 Base: CIFAR100-NIN HD-CNN, no finetuning HD-CNN, finetuning w/o CCC HD-CNN, finetuning w/ CCC 35.27 33.33 33.21 32.62 initial learning rate is 0.001 and is reduced by a factor of 10 once after 10K iterations. For evaluation, we use 10-view testing [18]. We extract five 26 × 26 patches (the 4 corner patches and the center patch) as well as their horizontal reflections and average their predictions. The CIFAR100-NIN net obtains 35.27% testing error. Our HD-CNN achieves testing error of 32.62% which improves the building block net by 2.65%. Category hierarchy. During the construction of the category hierarchy, the number of coarse categories can be adjusted by the clustering algorithm. We can also make the coarse categories either disjoint or overlapping by varying the hyperparameter γ. Thus we investigate their impacts on the classification error. We experiment with 5, 9, 14 and 19 coarse categories and vary the value of γ. The best results are obtained with 9 overlapping coarse categories and γ = 5 as shown in Fig 2 left. A histogram of fine category occurrences in 9 overlapping coarse categories is shown in Fig 2 right. The optimal value of coarse category number and hyperparameter γ are dataset dependent, mainly affected by the inherent hierarchy within the categories. 7. Experiments 7.1. Overview We evaluate HD-CNN on the benchmark datasets CIFAR100 [17] and ImageNet [4]. HD-CNN is implemented on the widely deployed Caffe [15] software. The network is trained by back propagation [18]. We run all the testing experiments on a single NVIDIA Tesla K40c card. 7.2. CIFAR100 Dataset The CIFAR100 dataset consists of 100 classes of natural images. There are 50K training images and 10K testing images. We follow [11] to preprocess the datasets (e.g. global contrast normalization and ZCA whitening). Randomly cropped and flipped image patches of size 26 × 26 are used for training. We adopt a NIN network 1 with three stacked layers [21]. We denote it as CIFAR100-NIN which will be the HD-CNN building block. Fine category components share preceding layers from conv1 to pool1 which accounts for 6% of the total parameters and 29% of the total floating point operations. The remaining layers are used as independent layers. For building the category hierarchy, we randomly choose 10K images from the training set as held-out set. Fine categories within the same coarse categories are visually more similar. We pretrain the rear layers of fine category components. The initial learning rate is 0.01 and it is decreased by a factor of 10 every 6K iterations. Fine-tuning is performed for 20K iterations with large mini-batches of size 256. The Disjoint coarse categories Overlapping coarse categories, γ = 5 Overlapping coarse categories, γ = 10 35.5 25 20 35 34.5 15 34 10 33.5 5 33 5 10 15 20 number of coarse categories 0 0 1 2 3 4 5 6 7 8 fine category occurrences Figure 2: Left: HD-CNN 10-view testing error against the number of coarse categories on CIFAR100 dataset. Right: Histogram of fine category occurrences in 9 overlapping coarse categories. 1 https://github.com/mavenlin/cuda-convnet/blob/ master/NIN/cifar-100_def Shared layers. The use of shared layers makes both com4325 putational complexity and memory footprint of HD-CNN sublinear in the number of fine category classifiers when compared to the building block net. Our HD-CNN with 9 fine category classifiers based on CIFAR100-NIN consumes less than three times as much memory as the building block net without parameter compression. We also want to investigate the impact of the use of shared layers on the classification error, memory footprint and the net execution time (Table 2). We build another HD-CNN where coarse category component and all fine category components use independent preceding layers initialized from a pretrained building block net. Under the single-view testing where only a central cropping is used, we observe a minor error increase from 34.36% to 34.50%. But using shared layers dramatically reduces the memory footprint from 1356 MB to 459 MB and testing time from 2.43 seconds to 0.28 seconds. Conditional executions. By varying the hyperparameter β, we can effectively affect the number of fine category components that will be executed. There is a trade-off between execution time and classification error. A larger value of β leads to higher accuracy at the cost of executing more components for fine categorization. By enabling conditional executions with hyperparameter β = 6, we obtain a substantial 2.8X speed up with merely a minor increase in error from 34.36% to 34.57% (Table 2). The testing time of HD-CNN is about 2.5 times as much as that of the building block net. Parameter compression. As fine category CNNs have independent layers from conv2 to cccp6, we compress them and reduce the memory footprint from 447MB to 286MB with a minor increase in error from 34.57% to 34.73%. Comparison with a strong baseline. As our HD-CNN memory footprint is about two times as much as the building block model (Table 2), it is necessary to compare a stronger baseline of similar complexity with HD-CNN. We adapt CIFAR100-NIN and double the number of filters in all convolutional layers which accordingly increases the memory footprint by three times. We denote it as CIFAR100-NINdouble and obtain error 34.26% which is 1.01% lower than that of the building block net but is 1.64% higher than that of HD-CNN. Comparison with model averaging. HD-CNN is fundamentally different from model averaging [18]. In model averaging, all models are capable of classifying the full set of the categories and each one is trained independently. The main sources of their prediction differences are different initializations. In HD-CNN, each fine category classifier only excels at classifying a partial set of categories. To compare HD-CNN with model averaging, we independently train two CIFAR100-NIN networks and take their averaged prediction as the final prediction. We obtain an error of 35.13%, which is about 2.51% higher than that of HD- CNN (Table 1). Note that HD-CNN is orthogonal to the model averaging and an ensemble of HD-CNN networks can further improve the performance. Coarse category consistency. To verify the effectiveness of coarse category consistency term in our loss function (4), we fine-tune a HD-CNN using the traditional multinomial logistic loss function. The testing error is 33.21%, which is 0.59% higher than that of a HD-CNN fine-tuned with coarse category consistency (Table 1). Comparison with state-of-the-art. Our HD-CNN improves on the current two best methods [19] and [30] by 2.06% and 1.16% respectively and sets new state-of-the-art results on CIFAR100 (Table 1). 7.3. ImageNet 1000-class Dataset The ILSVRC-2012 ImageNet dataset consists of about 1.2 million training images, 50, 000 validation images. To demonstrate the generality of HD-CNN, we experiment with two different building block nets. In both cases, HDCNNs achieve significantly lower testing errors than the building block nets. 7.3.1 Network-In-Network Building Block Net We choose a public 4-layer NIN net 2 as our first building block as it has greatly reduced number of parameters compared to AlexNet [18] but similar error rates. It is denoted as ImageNet-NIN. In HD-CNN, various components share preceding layers from conv1 to pool3 which account for 26% of the total parameters and 82% of the total floating point operations. We follow the training and testing protocols as in [18]. Original images are resized to 256 × 256. Randomly cropped and horizontally reflected 224 × 224 patches are used for training. At test time, the net makes a 10-view averaged prediction. We train ImageNet-NIN for 45 epochs. The top-1 and top-5 errors are 39.76% and 17.71%. To build the category hierarchy, we take 100K training images as the held-out set and find 89 overlapping coarse categories. Each fine category CNN is fine-tuned for 40K iterations while the initial learning rate 0.01 is decreased by a factor of 10 every 15K iterations. Fine-tuning the complete HD-CNN is not performed as the required minibatch size is significantly higher than that for the building block net. Nevertheless, we still achieve top-1 and top-5 errors of 36.66% and 15.80% and improve the building block net by 3.1% and 1.91%, respectively (Table 3). The classwise top-5 error improvement over the building block net is shown in Fig 4 left. Case studies We want to investigate how HD-CNN corrects the mistakes made by the building block net. In Fig 3, we 2 https://gist.github.com/mavenlin/ d802a5849de39225bcc6 4326 hermit crab hand blower digital clock ice lolly (a) alligator lizard scorpion crayfish fiddler crab tarantula 0 plunger barbell punching bag dumbbell power drill 0 odometer laptop projector cab notebook 0 bubble sunglasses sunglass cinema whistle 0 1 1 1 1 CC 6 CC 11 CC 10 CC 7 CC 82 0 CC 49 CC 74 CC 40 CC 81 CC 50 0 CC 65 CC 72 CC 75 CC 53 CC 63 0 CC 62 CC 75 CC 78 CC 50 CC 59 0 (b) 1 1 1 1 hermit crab alligator lizard fiddler crab banded gecko cricket 0 punching bag hand blower hair spray plunger barbell 0 iPod odometer pool table notebook digital clock 0 maypole umbrella ice lolly garbage truck scoreboard 0 1 1 1 1 (c) (d) tarantula ringneck snake hermit crab banded gecko alligator lizard 0 hand blower punching bag hair spray maraca barbell 0 monitor odometer laptop digital clock iPod 0 ice lolly sunglass sunglasses plunger maraca 0 hermit crab thunder snake scorpion gar banded gecko 1 0 punching bag dumbbell hair spray hand blower maraca 1 0 laptop digital clock Polaroid camera ski mask traffic light 1 0 ice lolly bow tie whistle maypole cinema 1 0 (e) 1 1 1 1 (f) hermit crab tarantula fiddler crab alligator lizard ringneck snake 0 hand blower punching bag hair spray plunger dumbbell 0 digital clock laptop odometer iPod projector 0 ice lolly sunglass sunglasses umbrella maypole 0 1 1 1 1 (g) Figure 3: Case studies on ImageNet dataset. Each row represents a testing case. Column (a): test image with ground truth label. Column (b): top 5 guesses from the building block net ImageNet-NIN. Column (c): top 5 Coarse Category (CC) probabilities. Column (d)-(f): top 5 guesses made by the top 3 fine category CNN components. Column (g): final top 5 guesses made by the HD-CNN. See text for details. Table 2: Comparison of testing errors, memory footprint and testing time between building block nets and HD-CNNs on CIFAR100 and ImageNet datasets. Statistics are collected under single-view testing. Three building block nets CIFAR100-NIN, ImageNet-NIN and ImageNet-VGG-16layer are used. The testing mini-batch size is 50. Notations: SL=Shared layers, CE=Conditional executions, PC=Parameter compression. Model top-1, top-5 Memory Time (MB) (sec.) Base:CIFAR100-NIN HD-CNN w/o SL HD-CNN HD-CNN+CE HD-CNN+CE+PC 37.29 34.50 34.36 34.57 34.73 188 1356 459 447 286 0.04 2.43 0.28 0.10 0.10 Base:ImageNet-NIN HD-CNN HD-CNN+CE HD-CNN+CE+PC 41.52, 18.98 37.92, 16.62 38.16, 16.75 38.39, 16.89 535 3544 3508 1712 0.19 3.25 0.52 0.53 Base:ImageNet-VGG16-layer HD-CNN+CE+PC 32.30, 12.74 4134 1.04 31.34, 12.26 6863 5.28 collect four testing cases. In the first case, the building block net fails to predict the label of the tiny hermit crab in the top 5 guesses. In HD-CNN, two coarse categories #6 and #11 receive most of the coarse probability mass. The fine category component #6 specializes in classifying crab breeds and strongly suggests the ground truth label. By combining the predictions from the top fine category classifiers, the HD-CNN predicts hermit crab as the most probable label. In the second case, the ImageNet-NIN confuses the ground truth hand blower with other objects of close shapes and appearances, such as plunger and barbell. For HD-CNN, the coarse category component is also not confident about which coarse category the object belongs to and thus assigns even probability mass to the top coarse categories. For the top 3 fine category classifiers, #74 strongly predicts ground truth label while the other two #49 and #40 rank the ground truth label at the 2nd and 4th place respectively. Overall, the HD-CNN ranks the ground truth label at the 1st place. This demonstrates HD-CNN needs to rely on multiple fine category classifiers to make correct predictions for difficult cases. Overlapping coarse categories.To investigate the impact of overlapping coarse categories on the classification, we train another HD-CNN with 89 fine category classifiers using disjoint coarse categories. It achieves top-1 and top-5 errors of 38.44% and 17.03% respectively, which is higher than those of the HD-CNN using overlapping coarse category hierarchy by 1.78% and 1.23% (Table 3). Conditional executions. By varying the hyperparameter β, we can control the number of fine category components that will be executed. There is a trade-off between execution time and classification error as shown in Fig 4 right. A larger value of β leads to lower error at the cost of more executed fine category components. By enabling conditional executions with hyperparameter β = 8, we obtain a substantial 6.3X speed up with merely a minor increase of single-view testing top-5 error from 16.62% to 16.75% (Table 2). With such speedup, the HD-CNN testing time is less than 3 times as much as that of the building block net. Parameter compression. We compress independent layers conv4 and cccp7 as they account for 60% of the parame- 4327 Table 3: Comparisons of 10-view testing errors between ImageNet-NIN and HD-CNN. Notation CC=Coarse category. Method top-1, top-5 Base:ImageNet-NIN Model averaging (3 base nets) HD-CNN, disjoint CC HD-CNN 39.76, 17.71 38.54, 17.11 38.44, 17.03 36.66, 15.80 Mean # of executed components error improvement 0.2 0.1 0.0 −0.1 −0.2 −0.3 0 200 400 class 600 800 1000 12 0.18 10 0.175 8 0.17 6 0.165 4 0.16 2 0.155 0 -5 -4 -3 -2 -1 0 log β 1 2 3 4 Top 5 error ters in ImageNet-NIN. Their parameter matrices are of size 1024 × 3456 and 1024 × 1024 and we use compression hyperparameters (s, k) = (3, 128) and (s, k) = (2, 256). The compression factors are 4.8 and 2.7. The compression decreases the memory footprint from 3508MB to 1712MB and merely increases the top-5 error from 16.75% to 16.89% under single-view testing (Table 2). Table 4: Errors on ImageNet validation set. 0.15 2 Figure 4: Left: Class-wise HD-CNN top-5 error improvement over the building block net. Right: Mean number of executed fine category classifiers and top-5 error against hyperparameter β on the ImageNet validation dataset. Comparison with model averaging. As the HD-CNN memory footprint is about three times as much as the building block net, we independently train three ImageNet-NIN nets and average their predictions. We obtain top-5 error 17.11% which is 0.6% lower than the building block but is 1.31% higher than that of HD-CNN (Table 3). 7.3.2 aged prediction is used as the final prediction. Please refer to [26] for more training and testing details. On ImageNet validation set, ImageNet-VGG-16-layer achieves top-1 and top-5 errors 24.79% and 7.50% respectively. We build a category hierarchy with 84 overlapping coarse categories. We implement multi-GPU training on Caffe by exploiting data parallelism [26] and train the fine category classifiers on two NVIDIA Tesla K40c cards. The initial learning rate is 0.001 and it is decreased by a factor of 10 every 4K iterations. HD-CNN fine-tuning is not performed. Due to large memory footprint of the building block net (Table 2), the HD-CNN with 84 fine category classifiers cannot fit into the memory directly. Therefore, we compress the parameters in layers fc6 and fc7 as they account for over 85% of the parameters. Parameter matrices in fc6 and fc7 are of size 4096 × 25088 and 4096 × 4096. Their compression hyperparameters are (s, k) = (14, 64) and (s, k) = (4, 256). The compression factors are 29.9 and 8 respectively. The HD-CNN obtains top-1 and top-5 errors 23.69% and 6.76% on ImageNet validation set and improves over ImageNet-VGG-16-layer by 1.1% and 0.74% respectively. Method top-1, top-5 GoogLeNet,multi-crop [31] VGG-19-layer, dense [26] VGG-16-layer+VGG-19-layer,dense N/A,7.9 24.8,7.5 24.0,7.1 Base:ImageNet-VGG-16-layer,dense HD-CNN,dense 24.79,7.50 23.69,6.76 Comparison with state-of-the-art. Currently, the two best nets on ImageNet dataset are GoogLeNet [31] (Table 4) and VGG 19-layer network [26]. Using multi-scale multicrop testing, a single GoogLeNet net achieves top-5 error 7.9%. With multi-scale dense evaluation, a single VGG 19layer net obtains top-1 and top-5 errors 24.8% and 7.5% and improves top-5 error of GoogLeNet by 0.4%. Our HDCNN decreases top-1 and top-5 errors of VGG 19-layer net by 1.11% and 0.74% respectively. Furthermore, HD-CNN slightly outperforms the results of averaging the predictions from VGG-16-layer and VGG-19-layer nets. VGG-16-layer Building Block Net The second building block net we use is a 16-layer CNN from [26]. We denote it as ImageNet-VGG-16-layer3 . The layers from conv1 1 to pool4 are shared and they account for 5.6% of the total parameters and 90% of the total floating number operations. The remaining layers are used as independent layers in coarse and fine category classifiers. We follow the training and testing protocols as in [26]. For training, we first sample a size S from the range [256, 512] and resize the image so that the length of short edge is S. Then a randomly cropped and flipped patch of size 224 × 224 is used for training. For testing, dense evaluation is performed on three scales {256, 384, 512} and the aver- 8. Conclusions and Future Work We demonstrated that HD-CNN is a flexible deep CNN architecture to improve over existing deep CNN models. We showed this empirically on both CIFAR-100 and ImageNet datasets using three different building block nets. As part of future work, we plan to extend HD-CNN architectures to those with more than 2 hierarchical levels and also verify our empirical results in a theoretical framework. 3 https://github.com/BVLC/caffe/wiki/Model-Zoo 4328 References [20] L.-J. Li, C. Wang, Y. Lim, D. M. Blei, and L. Fei-Fei. Building and using a semantivisual image hierarchy. In CVPR, 2010. [21] M. Lin, Q. Chen, and S. Yan. Network in network. CoRR, 2013. [22] B. Liu, F. Sadeghi, M. Tappen, O. Shamir, and C. Liu. Probabilistic label trees for efficient large scale image classification. In CVPR, 2013. [23] M. Marszalek and C. Schmid. Semantic hierarchies for visual object recognition. In CVPR, 2007. [24] M. Marszałek and C. Schmid. Constructing category hierarchies for visual recognition. In ECCV. 2008. [25] R. Salakhutdinov, A. Torralba, and J. Tenenbaum. Learning to share visual appearance for multiclass object detection. In CVPR, 2011. [26] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. [27] J. Sivic, B. C. Russell, A. Zisserman, W. T. Freeman, and A. A. Efros. Unsupervised discovery of visual object class hierarchies. In CVPR, 2008. [28] J. T. Springenberg and M. Riedmiller. Improving deep neural networks with probabilistic maxout units. arXiv preprint arXiv:1312.6116, 2013. [29] N. Srivastava and R. Salakhutdinov. Discriminative transfer learning with tree-based priors. In NIPS, 2013. [30] M. F. Stollenga, J. Masci, F. Gomez, and J. Schmidhuber. Deep networks with internal selective attention through feedback connections. In NIPS, 2014. [31] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. [32] A.-M. Tousch, S. Herbin, and J.-Y. Audibert. Semantic hierarchies for image annotation: A survey. Pattern Recognition, 2012. [33] N. Verma, D. Mahajan, S. Sellamanickam, and V. Nair. Learning hierarchical similarity metrics. In CVPR, 2012. [34] T. Xiao, J. Zhang, K. Yang, Y. Peng, and Z. Zhang. Errordriven incremental learning in deep convolutional neural network for large-scale image classification. In Proceedings of the ACM International Conference on Multimedia, 2014. [35] M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV. 2014. [36] M. D. Zeiler and R. Fergus. Stochastic pooling for regularization of deep convolutional neural networks. In ICLR, 2013. [37] B. Zhao, F. Li, and E. P. Xing. Large-scale category structure aware image categorization. In NIPS, 2011. [38] A. Zweig and D. Weinshall. Exploiting object hierarchy: Combining models from different category levels. In ICCV, 2007. [1] H. Bannour and C. Hudelot. Hierarchical image annotation using semantic hierarchies. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, 2012. [2] S. Bengio, J. Weston, and D. Grangier. Label embedding trees for large multi-class tasks. In NIPS, 2010. [3] J. Deng, N. Ding, Y. Jia, A. Frome, K. Murphy, S. Bengio, Y. Li, H. Neven, and H. Adam. Large-scale object classification using label relation graphs. In ECCV. 2014. [4] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. [5] J. Deng, J. Krause, A. C. Berg, and L. Fei-Fei. Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In CVPR, 2012. [6] J. Deng, S. Satheesh, A. C. Berg, and F. Li. Fast and balanced: Efficient label tree learning for large scale object recognition. In Advances in Neural Information Processing Systems, 2011. [7] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013. [8] R. Fergus, H. Bernal, Y. Weiss, and A. Torralba. Semantic label sharing for learning with many categories. In ECCV. 2010. [9] T. Gao and D. Koller. Discriminative learning of relaxed hierarchy for large-scale visual recognition. In ICCV, 2011. [10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. [11] I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In ICML, 2013. [12] G. Griffin and P. Perona. Learning and using taxonomies for fast visual categorization. In CVPR, 2008. [13] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV. 2014. [14] H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011. [15] Y. Jia. Caffe: An open source convolutional architecture for fast feature embedding, 2013. [16] Y. Jia, J. T. Abbott, J. Austerweil, T. Griffiths, and T. Darrell. Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies. In NIPS, 2013. [17] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 2009. [18] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. [19] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeplysupervised nets. arXiv preprint arXiv:1409.5185, 2014. 4329 Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture arXiv:1411.4734v4 [cs.CV] 17 Dec 2015 1 David Eigen1 Rob Fergus1,2 Dept. of Computer Science, Courant Institute, New York University 2 Facebook AI Research {deigen,fergus}@cs.nyu.edu Abstract lying in defining an appropriate training set and loss function; in this light, our work is a step towards building offthe-shelf regressor models that can be used for many applications. In addition, use of a single architecture helps simplify the implementation of systems that require multiple modalities, e.g. robotics or augmented reality, which in turn can help enable research progress in these areas. Lastly, in the case of depth and normals, much of the computation can be shared between modalities, making the system more efficient. In this paper we address three different computer vision tasks using a single multiscale convolutional network architecture: depth prediction, surface normal estimation, and semantic labeling. The network that we develop is able to adapt naturally to each task using only small modifications, regressing from the input image to the output map directly. Our method progressively refines predictions using a sequence of scales, and captures many image details without any superpixels or low-level segmentation. We achieve state-of-the-art performance on benchmarks for all three tasks. 2. Related Work Convolutional networks have been applied with great success for object classification and detection [19, 12, 30, 32, 34]. Most such systems classify either a single object label for an entire input window, or bounding boxes for a few objects in each scene. However, ConvNets have recently been applied to a variety of other tasks, including pose estimation [36, 27], stereo depth [39, 25], and instance segmentation [14]. Most of these systems use ConvNets to find only local features, or generate descriptors of discrete proposal regions; by contrast, our network uses both local and global views to predict a variety of output types. In addition, while each of these methods tackle just one or two tasks at most, we are able to apply our network to three disparate tasks. Our method builds upon the approach taken by Eigen et al. [8], who apply two convolutional networks in stages for single-image depth map prediction. We develop a more general network that uses a sequence of three scales to generate features and refine predictions to higher resolution, which we apply to multiple tasks, including surface normals estimation and per-pixel semantic labeling. Moreover, we improve performance in depth prediction as well, illustrating how our enhancements help improve all tasks. Single-image surface normal estimation has been addressed by Fouhey et al. [10, 11], Ladicky et al. [21], Barron and Malik [3, 2], and most recently by Wang et al. [38], the latter in work concurrent with ours. Fouhey et al. match to discriminative local templates [10] followed by a global op- 1. Introduction Scene understanding is a central problem in vision that has many different aspects. These include semantic labels describing the identity of different scene portions; surface normals or depth estimates describing the physical geometry; instance labels of the extent of individual objects; and affordances capturing possible interactions of people with the environment. Many of these are often represented with a pixel-map containing a value or label for each pixel, e.g. a map containing the semantic label of the object visible at each pixel, or the vector coordinates of the surface normal orientation. In this paper, we address three of these tasks, depth prediction, surface normal estimation and semantic segmentation — all using a single common architecture. Our multiscale approach generates pixel-maps directly from an input image, without the need for low-level superpixels or contours, and is able to align to many image details using a series of convolutional network stacks applied at increasing resolution. At test time, all three outputs can be generated in real time (∼30Hz). We achieve state-of-the art results on all three tasks we investigate, demonstrating our model’s versatility. There are several advantages in developing a general model for pixel-map regression. First, applications to new tasks may be quickly developed, with much of the new work 1 timization on a grid drawn from vanishing point rays [11], while Ladicky et al. learn a regression from over-segmented regions to a discrete set of normals and mixture coefficients. Barron and Malik [3, 2] infer normals from RGB-D inputs using a set of handcrafted priors, along with illumination and reflectance. From RGB inputs, Wang et al. [38] use convolutional networks to combine normals estimates from local and global scales, while also employing cues from room layout, edge labels and vanishing points. Importantly, we achieve as good or superior results with a more general multiscale architecture that can naturally be used to perform many different tasks. Prior work on semantic segmentation includes many different approaches, both using RGB-only data [35, 4, 9] as well as RGB-D [31, 29, 26, 6, 15, 17, 13]. Most of these use local features to classify over-segmented regions, followed by a global consistency optimization such as a CRF. By comparison, our method takes an essentially inverted approach: We make a consistent global prediction first, then follow it with iterative local refinements. In so doing, the local networks are made aware of their place within the global scene, and can can use this information in their refined predictions. Gupta et al. [13, 14] create semantic segmentations first by generating contours, then classifying regions using either hand-generated features and SVM [13], or a convolutional network for object detection [14]. Notably, [13] also performs amodal completion, which transfers labels between disparate regions of the image by comparing planes from the depth. Most related to our method in semantic segmentation are other approaches using convolutional networks. Farabet et al. [9] and Couprie et al. [6] each use a convolutional network applied to multiple scales in parallel generate features, then aggregate predictions using superpixels. Our method differs in several important ways. First, our model has a large, full-image field of view at the coarsest scale; as we demonstrate, this is of critical importance, particularly for depth and normals tasks. In addition, we do not use superpixels or post-process smoothing — instead, our network produces fairly smooth outputs on its own, allowing us to take a simple pixel-wise maximum. Pinheiro et al. [28] use a recurrent convolutional network in which each iteration incorporates progressively more context, by combining a more coarsely-sampled image input along with the local prediction from the previous iteration. This direction is precisely the reverse of our approach, which makes a global prediction first, then iteratively refines it. In addition, whereas they apply the same network parameters at all scales, we learn distinct networks that can specialize in the edits appropriate to their stage. Most recently, in concurrent work, Long et al. [24] adapt the recent VGG ImageNet model [32] to semantic segmen- Input Scale 1 !!!" conv/pool full conn. upsample Scale 2 !!!" concat conv/pool convolutions upsample Scale 3 !!!" concat conv/pool convolutions Depth Labels Normals Layer 1.1 Size 37x27 Scale 1 #convs 1 (AlexNet) #chan 96 ker. sz 11x11 Ratio /8 l.rate 0.001 Layer 1.1 Size 150x112 Scale 1 #convs 2 (VGG) #chan 64 ker. sz 3x3 Ratio /2 l.rate 0.001 1.2 18x13 1 256 5x5 /16 0.001 1.2 75x56 2 128 3x3 /4 0.001 1.3 18x13 1 384 3x3 /16 0.001 1.3 37x28 3 256 3x3 /8 0.001 1.4 18x13 1 384 3x3 /16 0.001 1.4 18x14 3 512 3x3 /16 0.001 1.5 1.6 1.7 upsamp 8x6 1x1 19x14 74x55 1 – – – 256 4096 64 64 3x3 – – – /32 – /16 /4 0.001 see text 1.5 1.6 1.7 upsamp 9x7 1x1 19x14 74x55 3 – – – 512 4096 64 64 3x3 – – – /32 – /16 /4 0.001 see text Layer Size #chan ker. sz Ratio l.rate 2.2 74x55 64 5x5 /4 0.01 2.3 74x55 64 5x5 /4 0.01 2.4 74x55 64 5x5 /4 0.01 2.5 74x55 C 5x5 /4 0.001 Scale 2 Scale 3 2.1 74x55 96+64 9x9 /4 0.001 Layer 3.1 3.2 3.3 3.4 Size 147x109 147x109 147x109 147x109 #chan 96+C 64 64 C ker. sz 9x9 5x5 5x5 5x5 Ratio /2 /2 /2 /2 l.rate 0.001 0.01 0.01 0.001 upsamp 147x109 C – /2 final 147x109 C – /2 Figure 1. Model architecture. C is the number of output channels in the final prediction, which depends on the task. The input to the network is 320x240. tation by applying 1x1 convolutional label classifiers at feature maps from different layers, corresponding to different scales, and averaging the outputs. By contrast, we apply networks for different scales in series, which allows them to make more complex edits and refinements, starting from the full image field of view. Thus our architecture easily adapts to many tasks, whereas by considering relatively smaller context and summing predictions, theirs is specific to semantic labeling. 3. Model Architecture Our model is a multi-scale deep network that first predicts a coarse global output based on the entire image area, then refines it using finer-scale local networks. This scheme is illustrated in Fig. 1. While our model was initially based upon the architecture proposed by [8], it offers several architectural improvements. First, we make the model deeper (more convolutional layers). Second, we add a third scale at higher resolution, bringing the final output resolution up to half the input, or 147 × 109 for NYUDepth. Third, instead of passing output predictions from scale 1 to scale 2, we pass multichannel feature maps; in so doing, we found we could also train the first two scales of the network jointly from the start, somewhat simplifying the training procedure and yielding performance gains. Scale 1: Full-Image View The first scale in the network predicts a coarse but spatially-varying set of features for the entire image area, based on a large, full-image field of view, which we accomplish this through the use of two fully-connected layers. The output of the last full layer is reshaped to 1/16-scale in its spatial dimensions by 64 features, then upsampled by a factor of 4 to 1/4-scale. Note since the feature upsampling is linear, this corresponds to a decomposition of a big fully connected layer from layer 1.6 to the larger 74 × 55 map; since such a matrix would be prohibitively large and only capable of producing a blurry output given the more constrained input features, we constrain the resolution and upsample. Note, however, that the 1/16-scale output is still large enough to capture considerable spatial variation, and in fact is twice as large as the 1/32-scale final convolutional features of the coarse stack. Since the top layers are fully connected, each spatial location in the output connects to the all the image features, incorporating a very large field of view. This stands in contrast to the multiscale approach of [6, 9], who produce maps where the field of view of each output location is a more local region centered on the output pixel. This full-view connection is especially important for depth and normals tasks, as we investigate in Section 7.1. As shown in Fig. 1, we trained two different sizes of our model: One where this scale is based on an ImageNettrained AlexNet [19], and one where it is initialized using the Oxford VGG network [32]. We report differences in performance between the models on all tasks, to measure the impact of model size in each. Scale 2: Predictions The job of the second scale is to produce predictions at a mid-level resolution, by incorporating a more detailed but narrower view of the image along with the full-image information supplied by the coarse network. We accomplish this by concatenating the feature maps of the coarse network with those from a single layer of convolution and pooling, performed at finer stride (see Fig. 1). The output of the second scale is a 55x74 prediction (for NYUDepth), with the number of channels depending on the task. We train Scales 1 and 2 of the model together jointly, using SGD on the losses described in Section 4. Scale 3: Higher Resolution The final scale of our model refines the predictions to higher resolution. We concatenate the Scale-2 outputs with feature maps generated from the original input at yet finer stride, thus incorporating a more detailed view of the image. The further refinement aligns the output to higher-resolution details, producing spatially coherent yet quite detailed outputs. The final output resolution is half the network input. 4. Tasks We apply this same architecture structure to each of the three tasks we investigate: depths, normals and semantic labeling. Each makes use of a different loss function and target data defining the task. 4.1. Depth For depth prediction, we use a loss function comparing the predicted and ground-truth log depth maps D and D∗ . Letting d = D − D∗ be their difference, we set the loss to !2 X 1 1X 2 ∗ d − 2 di Ldepth (D, D ) = n i i 2n i 1X + [(∇x di )2 + (∇y di )2 ] (1) n i where the sums are over valid pixels i and n is the number of valid pixels (we mask out pixels where the ground truth is missing). Here, ∇x di and ∇y di are the horizontal and vertical image gradients of the difference. Our loss is similar to that of [8], who also use the l2 and scale-invariant difference terms in the first line. However, we also include a first-order matching term (∇x di )2 + (∇y di )2 , which compares image gradients of the prediction with the ground truth. This encourages predictions to have not only close-by values, but also similar local structure. We found it indeed produces outputs that better follow depth gradients, with no degradation in measured l2 performance. 4.2. Surface Normals To predict surface normals, we change the output from one channel to three, and predict the x, y and z components of the normal at each pixel. We also normalize the vector at each pixel to unit l2 norm, and backpropagate through this normalization. We then employ a simple elementwise loss comparing the predicted normal at each pixel to the ground truth, using a dot product: 1X 1 Lnormals (N, N ∗ ) = − Ni · Ni∗ = − N · N ∗ (2) n i n where N and N ∗ are predicted and ground truth normal vector maps, and the sums again run over valid pixels (i.e. those with a ground truth normal). For ground truth targets, we compute the normal map using the same method as in Silberman et al. [31], which estimates normals from depth by fitting least-squares planes to neighboring sets of points in the point cloud. 4.3. Semantic Labels For semantic labeling, we use a pixelwise softmax classifier to predict a class label for each pixel. The final output then has as many channels as there are classes. We use a simple pixelwise cross-entropy loss, 1X ∗ C log(Ci ) (3) Lsemantic (C, C ∗ ) = − n i i P where Ci = ezi / c ezi,c is the class prediction at pixel i given the output z of the final convolutional linear layer 3.4. When labeling the NYUDepth RGB-D dataset, we use the ground truth depth and normals as additional input channels. We convolve each of the three input types (RGB, depth and normals) with a different set of 32 × 9 × 9 filters, then concatenate the resulting three feature sets along with the network output from the previous scale to form the input to the next. We also tried the “HHA” encoding proposed by [14], but did not see a benefit in our case, thus we opt for the simpler approach of using the depth and xyz-normals directly. Note the first scale is initialized using ImageNet, and we keep it RGB-only. Applying convolutions to each input type separately, rather than concatenating all the channels together in pixel space and filtering the joint input, enforces independence between the features at the lowest filter level, which we found helped performance. 5. Training 5.1. Training Procedure We train our model in two phases using SGD: First, we jointly train both Scales 1 and 2. Second, we fix the parameters of these scales and train Scale 3. Since Scale 3 contains four times as many pixels as Scale 2, it is expensive to train using the entire image area for each gradient step. To speed up training, we instead use random crops of size 74x55: We first forward-propagate the entire image through scales 1 and 2, upsample, and crop the resulting Scale 3 input, as well as the original RGB input at the corresponding location. The cropped image and Scale 2 prediction are forward- and back-propagated through the Scale 3 network, and the weights updated. We find this speeds up training by about a factor of 3, including the overhead for inference of the first two scales, and results in about the same if not slightly better error from the increased stochasticity. All three tasks use the same initialization and learning rates in nearly all layers, indicating that hyperparameter settings are in fact fairly robust to changes in task. Each were first tuned using the depth task, then verified to be an appropriate order of magnitude for each other task using a small validation set of 50 scenes. The only differences are: (i) The learning rate for the normals task is 10 times larger than depth or labels. (ii) Relative learning rates of layers 1.6 and 1.7 are 0.1 each for depth/normals, but 1.0 and 0.01 for semantic labeling. (iii) The dropout rate of layer 1.6 is 0.5 for depth/normals, but 0.8 for semantic labels, as there are fewer training images. We initialize the convolutional layers in Scale 1 using ImageNet-trained weights, and randomly initialize the fully connected layers of Scale 1 and all layers in Scales 2 and 3. We train using batches of size 32 for the AlexNet-initialized model but batches of size 16 for the VGG-initialized model due to memory constraints. In each case we step down the global learning rate by a factor of 10 after approximately 2M gradient steps, and train for an additional 0.5M steps. 5.2. Data Augmentation In all cases, we apply random data transforms to augment the training data. We use random scaling, in-plane rotation, translation, color, flips and contrast. When transforming an input and target, we apply corresponding transformations to RGB, depth, normals and labels. Note the normal vector transformation is the inverse-transpose of the worldspace transform: Flips and in-plane rotations require flipping or rotating the normals, while to scale the image by a factor s, we divide the depths by s but multiply the z coordinate of the normals and renormalize. 5.3. Combining Depth and Normals We combine both depths and normals networks together to share computation, creating a network using a single scale 1 stack, but separate scale 2 and 3 stacks. Thus we predict both depth and normals at the same time, given an RGB image. This produces a 1.6x speedup compared to using two separate models. 1 6. Performance Experiments 6.1. Depth We first apply our method to depth prediction on NYUDepth v2. We train using the entire NYUDepth v2 raw data distribution, using the scene split specified in the official train/test distribution. We then test on the common distribution depth maps, including filled-in areas, but constrained to the axis-aligned rectangle where there there is a valid depth map projection. Since the network output is a lower resolution than the original NYUDepth images, and excludes a small border, we bilinearly upsample our network outputs to the original 640x480 image scale, and extrapolate the missing border using a cross-bilateral filter. We compare our method to prior works Ladicky et al. [20], 1 This shared model also enabled us to try enforcing compatibility between predicted normals and those obtained via finite difference of the predicted depth (predicting normals directly performs considerably better than using finite difference). However, while this constraint was able to improve the normals from finite difference, it failed to improve either task individually. Thus, while we make use of the shared model for computational efficiency, we do not use the extra compatibility constraint. Surface Normal Estimation (GT [21]) 3DP [10] Ladicky &al. [21] Fouhey &al. [11] Wang &al. [38] Ours (AlexNet) Ours (VGG) Angle Distance Mean Median 35.3 31.2 33.5 23.1 35.2 17.9 26.9 14.8 23.7 15.5 20.9 13.2 Within t◦ Deg. 11.25◦ 22.5◦ 16.4 36.6 27.5 49.0 40.5 54.1 42.0 61.2 39.2 62.0 44.4 67.2 30◦ 48.2 58.7 58.9 68.2 71.1 75.9 Surface Normal Estimation (GT [31]) 3DP [10] Ladicky &al. [21] Wang &al. [38] Ours (AlexNet) Ours (VGG) Angle Distance Mean Median 37.7 34.1 35.5 25.5 28.8 17.9 25.9 18.2 22.2 15.3 Within t◦ Deg. 11.25◦ 22.5◦ 14.0 32.7 24.0 45.6 35.2 57.1 33.2 57.5 38.6 64.0 30◦ 44.1 55.9 65.5 67.7 73.9 Table 2. Surface normals prediction measured against the ground truth constructed by [21] (top) and [31] (bottom). (a) (b) (c) (d) Ladicky[20]Karsch[18] Baig [1] Liu [23] Eigen[8] Ours(A) Ours(VGG) δ < 1.25 0.542 – 0.597 0.614 0.614 0.697 0.769 δ < 1.252 0.829 – – 0.883 0.888 0.912 0.950 δ < 1.253 0.940 – – 0.971 0.972 0.977 0.988 abs rel – 0.350 0.259 0.230 0.214 0.198 0.158 sqr rel – – – – 0.204 0.180 0.121 RMS(lin) – 1.2 0.839 0.824 0.877 0.753 0.641 RMS(log) – – – – 0.283 0.255 0.214 sc-inv. – – 0.242 – 0.219 0.202 0.171 more details present. We measure performance with the same metrics as in [10]: The mean and median angle from the ground truth across all unmasked pixels, as well as the percent of vectors whose angle falls within three thresholds. Results are shown in Table 2. The smaller version of our model performs similarly or slightly better than Wang et al., while the larger version substantially outperforms all comparison methods. Figure 3 shows example predictions. Note the details captured by our method, such as the curvature of the blanket on the bed in the first row, sofas in the second row, and objects in the last row. Table 1. Depth estimation measurements. Note higher is better for top rows of the table, while lower is better for the bottom section. 6.3. Semantic Labels Figure 2. Example depth results. (a) RGB input; (b) result of [8]; (c) our result; (d) ground truth. Note the color range of each image is individually scaled. Depth Prediction Karsh et al. [18], Baig et al. [1], Liu et al. [23] and Eigen et al. [8]. The results are shown in Table 1. Our model obtains best performance in all metrics, due to our larger architecture and improved training. In addition, the VGG version of our model significantly outperforms the smaller AlexNet version, reenforcing the importance of model size; this is the case even though the depth task is seemingly far removed from the classification task with which the initial coarse weights were first trained. Qualitative results in Fig. 2 show substantial improvement in detail sharpness over [8]. 6.2. Surface Normals Next we apply our method to surface normals prediction. We compare against the 3D Primitives (3DP) and “Indoor Origami” works of Fouhey et al. [10, 11], Ladicky et al. [21], and Wang et al. [38]. As with the depth network, we used the full raw dataset for training, since ground-truth normal maps can be generated for all images. Since different systems have different ways of calculating ground truth normal maps, we compare using both the ground truth as constructed in [21] as well as the method used in [31]. The differences between ground truths are due primarily to the fact that [21] uses more aggressive smoothing; thus [21] tends to present flatter areas, while [31] is noisier but keeps 6.3.1 NYU Depth We finally apply our method to semantic segmentation, first also on NYUDepth. Because this data provides a depth channel, we use the ground-truth depth and normals as input into the semantic segmentation network, as described in Section 4.3. We evaluate our method on semantic class sets with 4, 13 and 40 labels, described in [31], [6] and [13], respectively. The 4-class segmentation task uses highlevel category labels “floor”, “structure”, “furniture” and “props”, while the 13- and 40-class tasks use different sets of more fine-grained categories. We compare with several recent methods, using the metrics commonly used to evaluate each task: For the 4- and 13-class tasks we use pixelwise and per-class accuracy; for the 40-class task, we also compare using the mean pixel-frequency weighted Jaccard index of each class, and the flat mean Jaccard index. Results are shown in Table 3. We decisively outperform the comparison methods on the 4- and 14-class tasks. In the 40-class task, our model outperforms Gupta et al. ’14 with both model sizes, and Long et al. with the larger size. Qualitative results are shown in Fig. 4. Even though our method does not use superpixels or any piecewise constant assumptions, it nevertheless tends to produce large constant regions most of the time. 4-Class Semantic Segmentation Couprie &al. [6] Khan &al. [15] Stuckler &al. [33] Mueller &al. [26] Gupta &al. ’13 [13] Ours (AlexNet) Ours (VGG) Pixel 64.5 69.2 70.9 72.3 78 80.6 83.2 Class 63.5 65.6 67.0 71.9 – 79.1 82.0 13-Class Semantic Couprie &al. [6] Wang &al. [37] Hermans &al. [17] Khan &al. [15] ∗ Ours (AlexNet) Ours (VGG) Pixel 52.4 – 54.2 58.3 70.5 75.4 Class 36.2 42.2 48.0 45.1 59.4 66.9 40-Class Semantic Segmentation Gupta&al.’13 [13] Gupta&al.’14 [14] Long&al. [24] Ours (AlexNet) Ours (VGG) Pix. Acc. 59.1 60.3 65.4 62.9 65.6 Per-Cls Acc. 28.4 35.1 46.1 41.3 45.1 Freq. Jaccard 45.6 47.0 49.5 47.6 51.4 Av. Jaccard 27.4 28.6 34.0 30.8 34.1 Table 3. Semantic labeling on NYUDepth v2 ∗ Khan&al. use a different overlapping label set. Sift Flow Semantic Segmentation Farabet &al. (1) [9] Farabet &al. (2) [9] Tighe &al. [35] Pinheiro &al. [28] Long &al. [24] Ours (AlexNet) (1) Ours (AlexNet) (2) Ours (VGG) (1) Ours (VGG) (2) Pix. Acc. 78.5 74.2 78.6 77.7 85.1 84.0 81.6 86.8 83.8 Per-Class Acc. 29.6 46.0 39.2 29.8 51.7 42.0 48.2 46.4 55.7 Freq. Jacc – – – – 76.1 73.7 71.3 77.9 74.7 Av. Jacc – – – – 39.5 33.1 32.6 38.8 37.6 Table 4. Semantic labeling on the Sift Flow dataset. (1) and (2) correspond to non-reweighted and class-reweighted versions of our model (see text). 6.3.2 Sift Flow We confirm our method can be applied to additional scene types by evaluating on the Sift Flow dataset [22], which contains images of outdoor cityscapes and landscapes segmented into 33 categories. We found no need to adjust convolutional kernel sizes or learning rates for this dataset, and simply transfer the values used for NYUDepth directly; however, we do adjust the output sizes of the layers to match the new image sizes. We compare against Tighe et al. [35], Farabet et al. [9], Pinheiro [28] and Long et al. [24]. Note that Farabet et al. train two models, using empirical or rebalanced class distributions by resampling superpixels. We train a more class-balanced version of our model by reweighting each class in the cross-entropy loss; we weight each pixel by αc = median f req/f req(c) where f req(c) is the number of pixels of class c divided by the total number of pixels in images where c is present, and median f req is the median of these frequencies. Results are in Table 4; we compare regular (1) and reweighted (2) versions of our model against comparison methods. Our smaller model substantially outperforms all but Long et al. , while our larger model performs similarly to Long et al. This demonstrates our model’s adaptability not just to different tasks but also different data. 6.3.3 Pascal VOC In addition, we also verify our method using Pascal VOC. Similarly to Long et al. [24], we train using the 2011 train- Pascal VOC Semantic Segmentation 2011 Validation 2011 Test 2012 Test Dai&al.[7] Long&al.[24] Chen&al.[5] Ours (VGG) Pix. Acc. Per-Cls Acc. Freq.Jacc Av.Jacc – – – – 90.3 75.9 83.2 62.7 – – – – 90.3 72.4 82.9 62.2 Av.Jacc – 62.7 – 62.5 Av.Jacc 61.8 62.2 71.6 62.6 Table 5. Semantic labeling on Pascal VOC 2011 and 2012. Contributions of Scales Depth Scale 1 only Scale 2 only Scales 1 + 2 Scales 1 + 2 + 3 Normals Pixelwise Error lower is better 0.218 29.7 0.290 31.8 0.216 26.1 0.198 25.9 4-Class 13-Class RGB+D+N RGB RGB+D+N RGB Pixelwise Accuracy higher is better 71.5 71.5 58.1 58.1 77.4 67.2 65.1 53.1 80.1 74.4 69.8 63.2 80.6 75.3 70.5 64.0 Table 6. Comparison of networks for different scales for depth, normals and semantic labeling tasks with 4 and 13 categories. Largest single contributing scale is underlined. ing set augmented with 8498 training images collected by Hariharan et al. [16], and evaluate using the 736 images from the 2011 validation set not also in the Hariharan extra set, as well as on the 2011 and 2012 test sets. We perform online data augmentations as in our NYUDepth and Sift Flow models, and use the same learning rates. Because these images have arbitrary aspect ratio, we train our model on square inputs, and scale the smaller side of each image to 256; at test time we apply the model with a stride of 128 to cover the image (two applications are usually sufficient). Results are shown in Table 5 and Fig. 5. We compare with Dai et al. [7], Long et al. [24] and Chen et al. [5]; the latter is a more recent work that augments a convolutional network with large top-layer field of and fullyconnected CRF. Our model performs comparably to Long et al., even as it generalizes to multiple tasks, demonstrated by its adeptness at depth and normals prediction. 7. Probe Experiments 7.1. Contributions of Scales We compare performance broken down according to the different scales in our model in Table 6. For depth, normals and 4- and 13-class semantic labeling tasks, we train and evaluate the model using just scale 1, just scale 2, both, or all three scales 1, 2 and 3. For the coarse scale-1-only prediction, we replace the last fully connected layer of the coarse stack with a fully connected layer that outputs directly to target size, i.e. a pixel map of either 1, 3, 4 or 13 channels depending on the task. The spatial resolution is the same as is used for the coarse features in our model, and is upsampled in the same way. We report the “abs relative difference” measure (i.e. |D− D∗ |/D∗ ) to compare depth, mean angle distance for normals, and pixelwise accuracy for semantic segmentation. First, we note there is progressive improvement in all tasks as scales are added (rows 1, 3, and 4). In addition, we find the largest single contribution to performance is the RGB input 3DP [10] Ladicky&al. [21] Wang&al. [38] Ours (VGG) Ground Truth Figure 3. Comparison of surface normal maps. Effect of Depth/Normals Inputs RGB only RGB + pred. D&N RGB + g.t. D&N Scale 2 only Pix. Acc. Per-class 53.1 38.3 58.7 43.8 65.1 52.3 Scales 1 + 2 Pix. Acc. Per-class 63.2 50.6 65.0 49.5 69.8 58.9 Table 7. Comparison of RGB-only, predicted depth/normals, and ground-truth depth/normals as input to the 13-class semantic task. coarse Scale 1 for depth and normals, but the more local Scale 2 for the semantic tasks — however, this is only due to the fact that the depth and normals channels are introduced at Scale 2 for the semantic labeling task. Looking at the labeling network with RGB-only inputs, we find that the coarse scale is again the larger contributer, indicating the importance of the global view. (Of course, this scale was also initialized with ImageNet convolution weights that are much related to the semantic task; however, even initializing randomly achieves 54.5% for 13-class scale 1 only, still the largest contribution, albeit by a smaller amount). 7.2. Effect of Depth and Normals Inputs The fact that we can recover much of the depth and normals information from the RGB image naturally leads to two questions: (i) How important are the depth and normals inputs relative to RGB in the semantic labeling task? (ii) What might happen if we were to replace the true depth and normals inputs with the predictions made by our network? To study this, we trained and tested our network using either Scale 2 alone or both Scales 1 and 2 for the 13class semantic labeling task under three input conditions: (a) the RGB image only, (b) the RGB image along with predicted depth and normals, or (c) RGB plus true depth and normals. Results are in Table 7. Using ground truth depth/normals shows substantial improvements over RGB alone. Predicted depth/normals appear to have little effect when using both scales, but a tangible improvement when using only Scale 2. We believe this is because any relevant information provided by predicted depths/normals for labeling can also be extracted from the input; thus the labeling network can learn this same information itself, just from the label targets. However, this supposes that the network structure is capable of learning these relations: If this is not the case, e.g. when using only Scale 2, we do see improvement. This is also consistent with Section 7.1, where we found the coarse network was important for prediction in all tasks — indeed, supplying the predicted depth/normals to scale 2 is able to recover much of the performance obtained by the RGB-only scales 1+2 model. 8. Discussion Together, depth, surface normals and semantic labels provide a rich account of a scene. We have proposed a simple and fast multiscale architecture using convolutional networks that gives excellent performance on all three modalities. The models beat existing methods on the vast majority of benchmarks we explored. This is impressive given that many of these methods are specific to a single modality and often slower and more complex algorithms than ours. As such, our model provides a convenient new baseline for the three tasks. To this end, code and trained models can be found at http://cs.nyu.edu/˜deigen/dnl/. RGB input 4-Class Prediction 13-Class Prediction 13-Class Ground Truth Figure 4. Example semantic labeling results for NYUDepth: (a) input image; (b) 4-class labeling result; (c) 13-class result; (d) 13-class ground truth. Figure 5. Example semantic labeling results for Pascal VOC 2011. For each image, we show RGB input, our prediction, and ground truth. Acknowledgements This work was supported by an ONR #N00014-13-1-0646 and an NSF CAREER grant. References [1] M. H. Baig and L. Torresani. Coarse-to-fine depth estimation from a single image via coupled regression and dictionary learning. arXiv:1501.04537, 2015. 5 [2] J. T. Barron and J. Malik. Intrinsic scene properties from a single rgb-d image. CVPR, 2013. 1, 2 [3] J. T. Barron and J. Malik. Shape, illumination, and reflectance from shading. TPAMI, 2015. 1, 2 [4] J. Carreira and C. Sminchisescu. Cpmc: Automatic object segmentation using constrained parametric min-cuts. PAMI, 2012. 2 [5] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR, 2015. 6 [6] C. Couprie, C. Farabet, L. Najman, and Y. LeCun. Indoor semantic segmentation using depth information. ICLR, 2013. 2, 3, 5, 6 [7] J. Dai, K. He, and J. Sun. Convolutional feature masking for joint object and stuff segmentation. arXiv 1412.1283, 2014. 6 [8] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. NIPS, 2014. 1, 3, 5 [9] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Scene parsing with multiscale feature learning, purity trees, and optimal covers. arXiv:1202.2160, 2012. 2, 3, 6 [10] D. F. Fouhey, A. Gupta, and M. Hebert. Data-driven 3d primitives for single image understanding. In ICCV, 2013. 1, 5, 7 [11] D. F. Fouhey, A. Gupta, and M. Hebert. Unfolding an indoor origami world. In ECCV, 2014. 1, 5 [12] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR, 2014. 1 [13] S. Gupta, P. Arbelaez, and J. Malik. Perceptual organization and recognition of indoor scenes from rgb-d images. In CVPR, 2013. 2, 5, 6 [14] S. Gupta, R. Girshick, P. Arbeláez, and J. Malik. Learning rich features from rgb-d images for object detection and segmentation. In ECCV. 2014. 1, 2, 4, 6 [15] S. K. Hameed, M. Bennamoun, F. Sohel, and R. Togneri. Geometry driven semantic labeling of indoor scenes. In ECCV. 2014. 2, 6 [16] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV. 2014. 6 [17] A. Hermans, G. Floros, and B. Leibe. Dense 3d semantic mapping of indoor scenes from rgb-d images. ICRA, 2014. 2, 6 [18] K. Karsch, C. Liu, S. B. Kang, and N. England. Depth extraction from video using non-parametric sampling. In TPAMI, 2014. 5 [19] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 1, 3 [20] L. Ladicky, J. Shi, and M. Pollefeys. Pulling things out of perspective. In CVPR, 2014. 4, 5 [21] L. Ladicky, B. Zeisl, and M. Pollefeys. Discriminatively trained dense surface normal estimation. In ECCV, 2014. 1, 5, 7 [22] C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. Freeman. Sift flow: dense correspondence across difference scenes. 2008. 6 [23] F. Liu, C. Shen, and G. Lin. Deep convolutional neural fields for depth estimation from a single image. arXiv:1411.6387, 2014. 5 [24] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. CoRR, abs/1411.4038, 2014. 2, 6 [25] R. Memisevic and C. Conrad. Stereopsis via deep learning. In NIPS Workshop on Deep Learning, 2011. 1 [26] A. C. Muller and S. Behnke. Learning depth-sensitive conditional random fields for semantic segmentation of rgb-d images. ICRA, 2014. 2, 6 [27] M. Osadchy, Y. Le Cun, and M. L. Miller. Synergistic face detection and pose estimation with energy-based models. In Toward Category-Level Object Recognition, pages 196–206. Springer, 2006. 1 [28] P. Pinheiro and R. Collobert. Recurrent convolutional neural networks for scene labeling. In ICML, 2014. 2, 6 [29] X. Ren, L. Bo, and D. Fox. Rgb-(d) scene labeling: Features and algorithms. In CVPR, 2012. 2 [30] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. ICLR, 2013. 1 [31] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 2, 4, 5 [32] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 1, 2, 3 [33] J. Stuckler, B. Waldvogel, H. Schulz, and S. Behnke. Dense real-time mapping of object-class semantics from rgb-d video. J. Real-Time Image Processing, 2014. 6 [34] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. 1 [35] J. Tighe and S. Lazebnik. Finding things: Image parsing with regions and per-exemplar detectors. In CVPR, 2013. 2, 6 [36] J. Tompson, A. Jain, Y. LeCun, and C. Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. NIPS, 2014. 1 [37] A. Wang, J. Lu, G. Wang, J. Cai, and T.-J. Cham. Multimodal unsupervised feature learning for rgb-d scene labeling. In ECCV, 2014. 6 [38] X. Wang, D. F. Fouhey and A. Gupta. Designing deep networks for surface normal estimation. In CVPR, 2015. 1, 2, 5, 7 [39] J. Zbontar and Y. LeCun. Computing the stereo matching cost with a convolutional neural network. CoRR, abs/1409.4326, 2014. 1 arXiv:1504.06201v3 [cs.CV] 21 Sep 2015 High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision Gedas Bertasius University of Pennsylvania Jianbo Shi University of Pennsylvania Lorenzo Torresani Dartmouth College gberta@seas.upenn.edu jshi@seas.upenn.edu lt@dartmouth.edu Abstract SotA HFL Most of the current boundary detection systems rely exclusively on low-level features, such as color and texture. However, perception studies suggest that humans employ object-level reasoning when judging if a particular pixel is a boundary. Inspired by this observation, in this work we show how to predict boundaries by exploiting objectlevel features from a pretrained object-classification network. Our method can be viewed as a High-for-Low approach where high-level object features inform the low-level boundary detection process. Our model achieves state-ofthe-art performance on an established boundary detection benchmark and it is efficient to run. Additionally, we show that due to the semantic nature of our boundaries we can use them to aid a number of highlevel vision tasks. We demonstrate that by using our boundaries we improve the performance of state-of-the-art methods on the problems of semantic boundary labeling, semantic segmentation and object proposal generation. We can view this process as a Low-for-High scheme, where lowlevel boundaries aid high-level vision tasks. Thus, our contributions include a boundary detection system that is accurate, efficient, generalizes well to multiple datasets, and is also shown to improve existing stateof-the-art high-level vision methods on three distinct tasks. Low-Level Task BD ODS 0.76 [27] 0.77 High-Level Tasks SS AP PI-IOU 19.9 [11] 45.8 [5] 54.6 48.8 SBL MF 28.0 [11] 62.5 OP MR 0.88 [31] 0.90 Table 1: Summary of results achieved by our proposed method (H F L) and state-of-the-art methods (SotA). We provide results on four vision tasks: Boundary Detection (BD), Semantic Boundary Labeling (SBL), Semantic Segmentation (SS), and Object Proposal (OP). The evaluation metrics include ODS F-score for BD task, max F-score (MF) and average precision (AP) for SBL task, per image intersection over union (PI-IOU) for SS task, and max recall (MR) for OP task. Our method produces better results in each of these tasks according to these metrics. ilar to how humans reason. Our boundary detection scheme can be viewed as a High-for-Low approach where we use high-level object features as cues for a low-level boundary detection process. Throughout the rest of the paper, we refer to our proposed boundaries as High-for-Low boundaries (H F L). We present an efficient deep network that uses objectlevel information to predict the boundaries. Our proposed architecture reuses features from the sixteen convolutional layers of the network of Simonyan et al. [29], which we refer to as VGG net. The VGG net has been trained for object classification, and therefore, reusing its features allows our method to utilize high-level object information to predict H F L boundaries. In the experimental section, we demonstrate that using object-level features produces semantically meaningful boundaries and also achieves above state-of-the-art boundary detection accuracy. Additionally, we demonstrate that we can successfully apply our H F L boundaries to a number of high-level vision tasks. We show that by using H F L boundaries we improve the results of three existing state-of-the-art methods on the tasks of semantic boundary labeling, semantic segmenta- 1. Introduction In the vision community, boundary detection has always been considered a low-level problem. However, psychological studies suggest that when a human observer perceives boundaries, object level reasoning is used [13, 25, 17]. Despite these findings, most of the boundary detection methods rely exclusively on low-level color and gradient features. In this work, we present a method that uses objectlevel features to detect boundaries. We argue that using object-level information to predict boundaries is more sim1 tion and object proposal generation. Therefore, using H F L boundaries to boost the results in high level vision tasks can be viewed as a Low-for-High scheme, where boundaries serve as low-level cues to aid high-level vision tasks. We present the summarized results for the boundary detection and the three mentioned high-level vision tasks in Table 1. Specifically, we compare our proposed method and an appropriate state-of-the-art method for that task. As the results indicate, we achieve better results in each of the tasks for each presented evaluation metric. We present more detailed results for each of these tasks in the later sections. In summary, our contributions are as follows. First, we show that using object-level features for boundary detection produces perceptually informative boundaries that outperform prior state-of-the-art boundary detection methods. Second, we demonstrate that we can use H F L boundaries to enhance the performance on the high-level vision tasks of semantic boundary labeling, semantic segmentation and object proposal. Finally, our method can detect boundaries in near-real time. Thus, we present a boundary detection system that is accurate, efficient, and is also applicable to high level vision tasks. 2. Related Work Most of the contour detection methods predict boundaries based purely on color, text, or other low-level features. We can divide these methods into three broad categories: spectral methods, supervised discriminative methods and deep learning based methods. Spectral methods formulate contour detection problem as an eigenvalue problem. The solution to this problem is then used to reason about the boundaries. The most successful approaches in this genre are the MCG detector [2], gPb detector [1], PMI detector [14], and Normalized Cuts [28]. Some of the notable discriminative boundary detection methods include sketch tokens (ST) [18], structured edges (SE) [6] and sparse code gradients (SCG) [23]. While SCG use supervised SVM learning [4], the latter two methods rely on a random forest classifier and models boundary detection as a classification task. Recently there have been attempts to apply deep learning to the task of boundary detection. SCT [20] is a sparse coding approach that reconstructs an image using a learned dictionary and then detect boundaries. Both N 4 fields [10] and DeepNet [16] approaches use Convolutional Neural Networks (CNNs) to predict edges. N 4 fields rely on dictionary learning and the use of the Nearest Neighbor algorithm within a CNN framework while DeepNet uses a traditional CNN architecture to predict contours. The most similar to our approach is DeepEdge [3], which uses a multi-scale bifurcated network to perform contour detection using object-level features. However, we show that our method achieves better results even without Input Image Upsampled Image - Candidate Contour Points ... |{z} 35x35x512 1100x1100x64 500x375 1100x1100 5504 Convolutional Feature Maps Predicted Boundaries Feature Interpolation & Concatenation |{z} 512 1024 500x375 5504-Dimensional Shared Weights for all Feature Vectors Candidate Points Figure 1: An illustration of our architecture (best viewed in color). First we extract a set of candidate contour points. Then we upsample the image and feed it through 16 convolutional layers pretrained for object classification. For each candidate point, we find its correspondence in each of the feature maps and perform feature interpolation. This yields a 5504-dimensional feature vector for each candidate point. We feed each of these vectors to two fully connected layers and store the predictions to produce a final boundary map. the complicated multi-scale and bifurcated architecture of DeepEdge. Additionally, unlike DeepEdge, our system can run in near-real time. In comparison to prior approaches, we offer several contributions. First, we propose to use object-level information to predict boundaries. We argue that such an approach leads to semantic boundaries, which are more consistent with humans reasoning. Second, we avoid feature engineering by learning boundaries from human-annotated data. Finally, we demonstrate excellent results for both low-level and high-level vision tasks. For the boundary detection task, our proposed H F L boundaries outperform all of the prior methods according to both F-score metrics. Additionally, we show that because H F L boundaries are based on objectlevel features, they can be used to improve performance in the high-level vision tasks of semantic boundary labeling, semantic segmentation, and object proposal generation. 3. Boundary Detection In this section, we describe our proposed architecture and the specific details on how we predict H F L boundaries using our method. The detailed illustration of our architecture is presented in Figure 1. Figure 2: A visualization of selected convolutional feature maps from VGG network (resized to the input image dimension). Because VGG was optimized for an object classification task, it produces high activation values on objects and their parts. 3.1. Selection of Candidate Contour Points We first extract a set of candidate contour points with a high recall. Due to its efficiency and high recall performance, we use the SE edge detector [6]. In practice, we could eliminate this step and simply try to predict boundaries at every pixel. However, selecting a set of initial candidate contour points, greatly reduces the computational cost. Since our goal is to build a boundary detector that is both accurate and efficient, we use these candidate points to speed up the computation of our method. 3.2. Object-Level Features After selecting candidate contour points, we up-sample the original input image to a larger dimension (for example 1100×1100). The up-sampling is done to minimize the loss of information due to the input shrinkage caused by pooling at the different layers. Afterwards, we feed the up-sampled image through 16 convolutional layers of the VGG net. We use the VGG net as our model because it has been trained to recognize a large number of object classes (the 1000 categories of the ImageNet dataset [24]) and thus encodes object-level features that apply to many classes. To preserve specific location information we utilize only the 16 convolutional layers of the VGG net. We don’t use fully connected layers because they don’t preserve spatial information, which is crucial for accurate boundary detection. We visualize some of the selected convolutional maps in Figure 2. Note the high activation values around the various objects in the images, which confirms our hypothesis that the VGG net encodes object specific information in its convolutional feature maps. 3.3. Feature Interpolation Similarly to [26, 12, 19], we perform feature interpolation in deep layers. After the up-sampled image passes through all 16 convolutional layers, for each selected candidate contour point we find its corresponding point in the feature maps. Due to the dimension differences in convo- lutional maps these correspondences are not exact. Thus we perform feature interpolation by finding the four nearest points and averaging their activation values. This is done in each of the 5504 feature maps. Thus, this results in a 5504-dimensional vector for each candidate point. We note that the interpolation of convolutional feature maps is the crucial component that enables our system to predict the boundaries efficiently. Without feature interpolation, our method would need to independently process the candidate edge points by analyzing a small image patch around each point, as for example done in DeepEdge [3] which feeds one patch at a time through a deep network. However, when the number of candidate points is large (e.g., DeepEdge considers about 15K points at each of 4 different scales), their patches overlap significantly and thus a large amount of computation is wasted by recalculating filter response values over the same pixels. Instead, we can compute the features for all candidate points with a single pass through the network by performing deep convolution over the entire image (i.e., feeding the entire image rather than one patch at a time) and then by interpolating the convolutional feature maps at the location of each candidate edge point so as to produce its feature descriptor. Thanks to this speedup, our method has a runtime of 1.2 seconds (using a K40 GPU), which is better than the runtimes of prior deep-learning based edge detection methods [27, 10, 16, 3]. 3.4. Learning to Predict Boundaries After performing feature interpolation, we feed the 5504-dimensional feature vectors corresponding to each of the candidate contour points to two fully connected layers that are optimized to the human agreement criterion. To be more precise, we define our prediction objective as a fraction of human annotators agreeing on the presence of the boundary at a particular pixel. Therefore, a learning objective aims at mimicking the judgement of the human labelers. Finally, to detect H F L boundaries, we accumulate the predictions from the fully connected layers for each of the candidate points and produce a boundary probability map as illustrated in Figure 1. 3.5. Implementation Details In this section, we describe the details behind the training procedure of our model. We use the Caffe library [15] to implement our network architecture. In the training stage, we freeze the weights in all of the convolutional layers. To learn the weights in the two fully connected layers we train our model to optimize the least squares error of the regression criterion that we described in the previous subsection. To enforce regularization we set a dropout rate of 0.5 in the fully connected layers. Our training dataset includes 80K points from the BSDS500 dataset [22]. As described in the previous subsection, the labels represent the fraction of human annotators agreeing on the boundary presence. We divide the label space into four quartiles, and select an equal number of samples for each quartile to balance the training dataset. In addition to the training dataset, we also sample a hold-out dataset of size 40, 000. We use this for the hard-positive mining [21] in order to reduce the number of false-negative predictions. For the first 25 epochs we train the network on the original 80, 000 training samples. After the first 25 epochs, we test the network on the hold-out dataset and detect false negative predictions made by our network. We then augment the original 80, 000 training samples with the false negatives and the same number of randomly selected true negatives. For the remaining 25 epochs, we train the network on this augmented dataset. Consensus GT SCG [23] DeepNet [16] PMI [14] DeepEdge [3] N 4 -fields [10] HFL ODS 0.6 0.61 0.61 0.62 0.64 0.65 OIS 0.64 0.64 0.68 0.64 0.67 0.68 AP 0.56 0.61 0.56 0.64 0.64 0.67 FPS 1/280 1/5‡ 1/900 1/1000‡ 1/6‡ 5/6‡ Any GT SE [6] MCG [2] N 4 -fields [10] DeepEdge [3] MSC [30] DeepContour [27] HFL ODS 0.75 0.75 0.75 0.75 0.76 0.76 0.77 OIS 0.77 0.78 0.77 0.77 0.78 0.77 0.79 AP 0.80 0.76 0.78 0.81 0.79 0.8 0.8 FPS 2.5 1/24 1/6‡ 1/1000‡ 1/30‡ 5/6‡ Table 2: Boundary detection results on BSDS500 benchmark. Upper half of the table illustrates the results for “consensus” ground-truth criterion while the lower half of the table depicts the results for “any” ground-truth criterion. In both cases, our method outperforms all prior methods according to both ODS (optimal dataset scale) and OIS (optimal image scale) metrics. We also report the run-time of our method (‡ GPU time) in the FPS column (frames per second), which shows that our algorithm is faster than prior approaches based on deep learning [27, 10, 16, 3]. 3.6. Boundary Detection Results In this section, we present our results on the BSDS500 dataset [22], which is the most established benchmark for boundary detection. The quality of the predicted boundaries is evaluated using three standard measures: fixed contour threshold (ODS), per-image best threshold (OIS), and average precision (AP). We compare our approach to the state-of-the-art based on two different sets of BSDS500 ground truth boundaries. First, we evaluate the accuracy by matching each of the predicted boundary pixels with the ground truth boundaries that were annotated by any of the human annotators. This set of ground truth boundaries is referred to as “any”. We present the results for “any” ground truth boundaries in the lower half of Table 2. As indicated by the results, H F L boundaries outperform all the prior methods according to both F-score measures. Recently, there has been some criticism raised about the procedure for boundary detection evaluation on the BSDS500 dataset. One issue with the BSDS500 dataset involves the so called “orphan” boundaries: the bound- Figure 5: Qualitative results on the BSDS benchmark. The first column of images represent input images. The second column illustrates SE [6], while the third column depicts H F L boundaries. Notice that SE boundaries are predicted with low confidence if there is no significant change in color between the object and the background. Instead, because our model is defined in terms of object-level features, it can predict object boundaries with high confidence even if there is no significant color variation in the scene. aries that are marked by only one or two human annotators. These “orphan” boundaries comprise around 30% of BSDS500 dataset but most of them are considered uninformative. However, the standard evaluation benchmark rewards the methods that predict these boundaries. To resolve 4. High-Level Vision Applications Weight Magnitude −3 4 x 10 2 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Feature Map Number (Sorted in the Order from Early to Deep Layers) 5000 5500 Figure 6: We train a linear regression model and visualize its weight magnitudes in order to understand which features are used most heavily in the boundary prediction (this linear regression is used only for the visualization purposes and not for the accuracy analysis). Note how heavily weighted features lie in the deepest layers of the network, i.e., the layers that are most closely associated with object information. this issue we also evaluate our H F L boundaries on the so called “consensus” set of ground truth boundaries. These “consensus” boundaries involve only boundaries that are marked by all of the human annotators and hence, are considered perceptually meaningful. In the upper half of Table 2, we present the results achieved by our method on the “consensus” set of the ground truth boundaries. Our H F L boundaries outperform or tie all the prior methods in each of the three evaluation metrics, thus suggesting that H F L boundaries are similar to the boundaries that humans annotated. We also report the runtimes in Table 2 and note that our method runs faster than previous deep-learning based edge detection systems [27, 10, 16, 3]. Our proposed model computes a highly nonlinear function of the 5504-dimensional feature vector of each candidate point. Thus, it is difficult to assess which features are used most heavily by our edge predictor. However, we can gain a better insight by replacing the nonlinear function with a simple linear model. In Fig. 6 we show the weight magnitudes of a simple linear regression model (we stress that this linear model is used only for feature visualization purposes). From this Figure, we observe that many important features are located in the deepest layers of the VGG network. As shown in [7], these layers encode high-level object information, which confirms our hypothesis that highlevel information is useful for boundary detection. Finally, we present some qualitative results achieved by our method in Figure 5. These examples illustrate the effective advantage that H F L boundaries provide over another state-of-the-art edge detection system, the SE system [6]. Specifically, observe the parts of the image where there is a boundary that separates an object from the background but where the color change is pretty small. Notice that because the SE boundary detection is based on low-level color and texture features, it captures these boundaries with very low confidence. In comparison, because H F L boundaries rely on object-level features, it detects these boundaries with high confidence. In this section, we describe our proposed Low-for-High pipeline: using low-level boundaries to aid a number of high-level vision tasks. We focus on the tasks of semantic boundary labeling, semantic segmentation and object proposal generation. We show that using H F L boundaries improves the performance of state-of-the-art methods in each of these high-level vision tasks. 4.1. Semantic Boundary Labeling The task of semantic boundary labeling requires not only to predict the boundaries but also to associate a specific object class to each of the boundaries. This implies that given our predicted boundaries we also need to label them with object class information. We approach this problem by adopting the ideas from the recent work on Fully Convolutional Networks (FCN) [19]. Given an input image, we concurrently feed it to our boundary-predicting network (described in Section 3), and also through the FCN that was pretrained for 20 Pascal VOC classes and the background class. While our proposed network produces H F L boundaries, the FCN model predicts class probabilities for each of the pixels. We can then merge the two output maps as follows. For a given boundary point we consider a 9 × 9 grid around that point from each of the 21 FCN object-class probability maps. We calculate the maximum value inside each grid, and then label the boundary at a given pixel with the object-class that corresponds to the maximum probability across these 21 maps. We apply this procedure for each of the boundary points, in order to associate object-class labels to the boundaries. Note that we consider the grids around the boundary pixel because the output of the FCN has a poor localization, and considering the grids rather than individual pixels leads to higher accuracy. We can also merge H F L boundaries with the state-ofthe-art DeepLab-CRF segmentation [5] to obtain higher accuracy. We do this in a similar fashion as just described. First, around a given boundary point we extract a 9 × 9 grid from the DeepLab-CRF segmentation. We then compute the mode value in the grid (excluding the background class), and use the object-class corresponding to the mode value as a label for the given boundary point. We do this for each of the boundary points. By merging H F L boundaries and the output of FCN or DeepLab-CRF, we get semantic boundaries that are highly localized and also contain objectspecific information. 4.1.1 Semantic Boundary Labeling Results In this section, we present semantic boundary labeling results on the SBD dataset [11], which includes ground truth boundaries that are also labeled with one of 20 Pascal VOC classes. The boundary detection accuracy for each class is Method (Metric) aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train InvDet (MF) H F L-FC8 (MF) H F L-CRF (MF) InvDet (AP) H F L-FC8 (AP) H F L-CRF (AP) 40.8 58.8 61.6 31.3 51.0 56.8 22.6 69.3 71.9 17.3 59.0 65.7 42.6 71.6 73.9 38.4 66.0 71.2 49.5 59.6 61.4 29.6 50.7 55.2 15.7 68.0 74.6 9.6 58.9 69.3 16.8 54.1 57.2 9.9 40.6 45.7 36.7 57.2 58.8 24.2 47.1 48.9 43.0 68.0 70.4 33.6 62.9 71.1 18.1 43.3 46.5 10.7 25.6 29.1 26.6 65.8 72.3 16.4 54.6 65.9 10.2 33.3 36.2 3.7 15.3 17.7 18.0 67.9 71.1 12.1 57.8 64.5 35.2 67.5 73.0 28.5 57.3 68.3 29.4 62.2 68.1 20.4 55.9 64.7 48.2 69.0 70.3 45.7 62.2 65.9 14.3 43.8 44.4 7.6 27.5 29.1 26.8 68.5 73.2 16.1 55.6 66.5 11.2 33.9 42.6 5.7 18.0 25.7 22.2 57.7 62.4 14.6 50.1 60.0 tv mean 32.0 54.8 60.1 22.7 40.6 49.8 28.0 58.7 62.5 19.9 47.8 54.6 Table 3: Results of semantic boundary labeling on the SBD benchmark using the Max F-score (MF) and Average Precision (AP) metrics. Our method (H F L) outperforms Inverse Detectors [11] for all 20 categories according to both metrics. Note that using the CRF output to label the boundaries produces better results than using the outputs from the FC8 layer of FCN. evaluated using the maximum F-score (MF), and average precision (AP) measures. Labeling boundaries with the semantic object information is a novel and still relatively unexplored problem. Therefore, we found only one other approach (Inverse Detectors) that tried to tackle this problem [11]. The basic idea behind Inverse Detectors consists of several steps. First, generic boundaries in the image are detected. Then, a number of object proposal boxes are generated. These two sources of information are then used to construct the features. Finally, a separate classifier is used to label the boundaries with the object-specific information. Table 3 shows that our approach significantly outperforms Inverse Detectors according to both the maximum Fscore and the average precision metrics for all twenty categories. As described in Section 4.1 we evaluate the two variants of our method. Denoted by H F L-FC8 is the variant for which we label H F L boundaries with the outputs from the last layer (FC8) of the pretrained FCN. We denote with H F L-CRF the result of labeling our boundaries with the output from the DeepLab-CRF [5]. Among these two variants, we show that the latter one produces better results. This is expected since the CRF framework enforces spatial coherence in the semantic segments. In Figure 7, we present some of the qualitative results produced by our method. We note that even with multiple objects in the image, our method successfully recognizes and localizes boundaries of each of the classes. 4.2. Semantic Segmentation For the semantic segmentation task, we propose to enhance the DeepLab-CRF [5] with our predicted H F L boundaries. DeepLab-CRF is a system comprised of a Fully Convolutional Network (described in Section 4.1) and a dense CRF applied on top of FCN predictions. Specifically, in the CRF, the authors propose to use a Gaussian kernel and a bilateral term including position and color terms as the CRF features (see [5]). While in most cases the proposed scheme works well, DeepLab-CRF sometimes produces segmentations that are not spatially coherent, particularly for images containing small object re- Figure 7: A visualization of the predicted semantic boundary labels. Images in the first column are input examples. Columns two and three show semantic H F L boundaries of different object classes. Note that even with multiple objects appearing simultaneously, our method outputs precise semantic boundaries. gions. We propose to address this issue by adding features based on our predicted H F L boundaries in the CRF framework. Note that we use predicted boundaries from Section 3 and not the boundaries labeled with the object information that we obtained in Section 4.1. We use the Normalized Cut [28] framework to generate our features. First, we construct a pixel-wise affinity matrix W using our H F L boundaries. We measure the similarity between two pixels as: Wij = exp (− max{ p∈ij M (p)2 }) σ2 where Wij represents the similarity between pixels i and j, p denotes the boundary point along the line segment ij connecting pixels i and j, M depicts the magnitude of the boundary at pixel p, and σ denotes the smoothness parameter, which is usually set to 14% of the maximum boundary value in the image. The intuitive idea is that two pixels are similar (i.e. Wij = 1) if there is no boundary crossing the line connecting these two pixels (i.e. M (p) = 0 ∀p ∈ ij) or if the boundary strength is low. We note that it is not necessary to build a full affinity matrix W . We build a sparse affin- Metric Method (Dataset) DL-CRF (VOC) DL-CRF+H F L (VOC) PP-IOU DL-CRF (SBD) DL-CRF+H F L (SBD) DL-CRF (VOC) DL-CRF+H F L (VOC) PI-IOU DL-CRF (SBD) DL-CRF+H F L (SBD) aero 78.6 77.9 74.2 75.1 46.1 47.5 59.4 63.4 bike 41.1 41.2 68.0 69.2 28.0 27.6 36.5 42.5 bird 83.5 83.1 81.9 81.6 48.5 50.4 58.0 58.4 boat 75.3 74.4 64.6 64.8 54.5 63.5 38.6 41.3 bottle 72.9 73.2 71.8 71.3 45.5 47.7 32.0 32.5 bus 83.1 85.5 86.3 86.4 57.6 57.9 58.1 61.2 car 76.6 76.1 78.3 78.1 34.1 38.7 44.7 45.7 cat 80.8 80.6 84.3 84.1 47.3 47.2 59.6 61.4 chair 37.8 35.7 41.6 41.2 19.5 21.1 25.8 28.4 cow 72.1 71.0 78.0 77.8 61.4 57.3 51.8 55.5 table 66.5 66.6 49.9 50.4 41.6 41.2 28.1 31.5 dog 64.7 64.3 82.0 81.6 42.5 43.7 59.0 61.4 horse 65.8 65.9 78.5 78.2 34.4 36.0 46.9 51.8 mbike person plant 75.7 80.5 34.4 75.2 80.2 32.8 77.1 80.1 54.3 78.5 80.7 53.8 61.8 62.1 22.1 66.4 61.1 21.3 50.3 61.8 22.2 54.6 62.1 24.9 sheep 75.9 75.2 75.6 74.9 50.5 53.9 45.9 52.6 sofa 47.4 47.0 49.8 49.1 41.0 42.1 33.4 34.2 train 86.6 87.1 79.5 79.5 61.2 70.9 62.1 67.1 tv 77.9 77.9 70.1 70.4 31.9 34.6 41.0 45.1 mean 68.9 68.5 71.4 71.4 44.6 46.5 45.8 48.8 Table 4: Semantic segmentation results on the SBD and VOC 2007 datasets. We measure the results according to PP-IOU (per pixel) and PI-IOU (per image) evaluation metrics. We denote the original DeepLab-CRF system and our proposed modification as DL-CRF and DL-CRF+H F L, respectively. According to the PP-IOU metric, our proposed features (DLCRF+H F L) yield almost equivalent results as the original DeepLab-CRF system. However, based on PI-IOU metric, our proposed features improve the mean accuracy by 3% and 1.9% on SBD and VOC 2007 datasets respectively. ity matrix connecting every pair of pixels i and j that have distance 5 or less from each other. After building a boundary-based affinity matrix W we P set Dii = i6=j Wij and compute eigenvectors v of the generalized eigenvalue system: (D − W)v = λDv We then resize the eigenvectors v to the original image dimensions, and use them as additional features to the CRF part of DeepLab-CRF system. In our experiments, we use the 16 eigenvectors corresponding to the smallest eigenvalues, which results in 16 extra feature channels. Note that the eigenvectors contain soft segmentation information. Because H F L boundaries predict object-level contours with high confidence, the eigenvectors often capture regions corresponding to objects. We visualize a few selected eigenvectors in Figure 8. In the experimental section, we demonstrate that our proposed features make the output produced by DeepLab-CRF more spatially coherent and improve the segmentation accuracy according to one of the metrics. We also note that our proposed features are applicable to any generic method that incorporates CRF. For instance, even if DeepLab-CRF used an improved DeepLab network architecture, our features would still be beneficial because they contribute directly to the CRF part and not the DeepLab network part of the system. 4.2.1 Semantic Segmentation Results In this section, we present semantic segmentation results on the SBD [11] and also Pascal VOC 2007 [8] datasets, which both provide ground truth segmentations for 20 Pascal VOC classes. We evaluate the results in terms of two metrics. The first metric measures the accuracy in terms of pixel intersection-over-union averaged per pixel (PP-IOU) across the 20 classes. According to this metric, the accuracy is computed on a per pixel basis. As a result, the images that contain large object regions are given more importance. Figure 8: In this figure, the first column depicts an input image while the second and third columns illustrate two selected eigenvectors for that image. The eigenvectors contain soft segmentation information. Because H F L boundaries capture object-level boundaries, the resulting eigenvectors primarily segment regions corresponding to the objects. We observe that while DeepLab-CRF works well on the images containing large object regions, it produces spatially disjoint outputs for the images with smaller and object regions (see Figure 9). This issue is often being overlooked, because according to the PP-IOU metric, the images with large object regions are given more importance and thus contribute more to the accuracy. However, certain applications may require accurate segmentation of small objects. Therefore, in addition to PP-IOU, we also consider the PIIOU metric (pixel intersection-over-union averaged per image across the 20 classes), which gives equal weight to each of the images. For both of the metrics we compare the semantic segmentation results of a pure DeepLab-CRF [5] and also a modification of DeepLab-CRF with our proposed features added to the CRF framework. We present the results for both of the metrics in Table 4. Based on these results, we observe that according to the first metric (PP-IOU), our proposed features yield almost equivalent results as the original DeepLab-CRF system. However, according to the second metric (PI-IOU) our features yield an average improvement of 3% and 1.9% in Figure 9: An illustration of the more challenging semantic segmentation examples. The first column depicts the predictions achieved by DeepLab-CRF, while the second column illustrates the results after adding our proposed features to the CRF framework. The last column represents ground truth segmentations. Notice how our proposed features render the predicted semantic segments more spatially coherent and overall more accurate. SBD and VOC 2007 datasets respectively. We also visualize the qualitative results produced by both approaches in Figure 9. Notice how our proposed features make the segmentations look smoother relative to the segmentations produced by the original DeepLab-CRF system. Once again, we want to stress that our H F L features are applicable to any method that uses the CRF. Therefore, based on the results presented in this section, we believe that our proposed features could be beneficial in a wide array of problems that involve the use of the CRF framework. 4.3. Object Proposals Finally, we show that our method produces object-level boundaries that can be successfully exploited in an object proposal scheme. Specifically we adopt the EdgeBoxes approach [31], which can be applied to any generic boundaries to produce a list of object proposal boxes. The original EdgeBoxes method uses SE boundaries to generate the boxes. However, SE boundaries are predicted using lowlevel color and texture features, rather than object-level features. Thus, here we validate the hypothesis that the EdgeBoxes proposals can be improved by replacing the SE boundaries with our H F L boundaries. 4.3.1 Object Proposal Results In this section, we present object proposal results on the Pascal VOC 2012 dataset [9]. We evaluate the quality of bounding-box proposals according to three metrics: area under the curve (AUC), the number of proposals needed to reach recall of 75%, and the maximum recall over 5000 object bounding-boxes. Additionally, we compute the accuracy for each of the metrics for three different intersection Method IoU 0.65 IoU 0.7 IoU 0.75 AUC N@75% Recall AUC N@75% Recall AUC N@75% Recall SE HFL 0.52 0.53 413 365 0.93 0.95 0.47 0.48 658 583 0.88 0.9 0.41 0.41 inf 2685 0.75 0.77 Table 5: Comparison of object proposal results. We compare the quality of object proposals using Structured Edges [6] and H F L boundaries. We evaluate the performance for three different IOU values and demonstrate that using H F L boundaries produces better results for each evaluation metric and for each IOU value. over union (IOU) values: 0.65, 0.7, and 0.75. We present these results in Table 5. As described in Section 4.3, we use EdgeBoxes [31], a package that uses generic boundaries, to generate object proposals. We compare the quality of the generated object proposals when using SE boundaries and H F L boundaries. We demonstrate that for each IOU value and for each of the three evaluation metrics, H F L boundaries produce better or equivalent results. This confirms our hypothesis that H F L boundaries can be used effectively for high-level vision tasks such as generating object proposals. 5. Conclusions In this work, we presented an efficient architecture that uses object-level information to predict semantically meaningful boundaries. Most prior edge detection methods rely exclusively on low-level features, such as color or texture, to detect the boundaries. However, perception studies suggest that humans employ object-level reasoning when deciding whether a given pixel is a boundary [13, 25, 17]. Thus, we propose a system that focuses on the semantic objectlevel cues rather than low level image information to detect the boundaries. For this reason we refer to our boundary detection scheme as a High-for-Low approach, where highlevel object features inform the low-level boundary detection process. In this paper we demonstrated that our proposed method produces boundaries that accurately separate objects and the background in the image and also achieve higher F-score compared to any prior work. Additionally, we showed that, because H F L boundaries are based on object-level features, they can be employed to aid a number of high level vision tasks in a Low-for-High fashion. We use our boundaries to boost the accuracy of state-of-the-art methods on the high-level vision tasks of semantic boundary labeling, semantic segmentation, and object proposals generation. We show that using H F L boundaries leads to better results in each of these tasks. To conclude, our boundary detection method is accurate, efficient, applicable to a variety of datasets, and also useful for multiple high-level vision tasks. We plan to release the source code for H F L upon the publication of the paper . 6. Acknowledgements We thank Mohammad Haris Baig for the suggestions and help with the software. This research was funded in part by NSF award CNS-1205521. References [1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 33(5):898–916, May 2011. 2 [2] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marqués, and J. Malik. Multiscale combinatorial grouping. In Computer Vision and Pattern Recognition (CVPR), 2014. 2, 4 [3] G. Bertasius, J. Shi, and L. Torresani. Deepedge: A multiscale bifurcated deep network for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015. 2, 3, 4, 5 [4] C. J. C. Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2:121–167, 1998. 2 [5] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR, abs/1412.7062, 2014. 1, 5, 6, 7 [6] P. Dollár and C. L. Zitnick. Fast edge detection using structured forests. PAMI, 2015. 2, 3, 4, 5, 8 [7] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. CoRR, abs/1310.1531, 2013. 5 [8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html. 7 [9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html. 8 [10] Y. Ganin and V. S. Lempitsky. N4 -fields: Neural network nearest neighbor fields for image transforms. ACCV, 2014. 2, 3, 4, 5 [11] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In International Conference on Computer Vision (ICCV), 2011. 1, 5, 6, 7 [12] B. Hariharan, P. A. Arbeláez, R. B. Girshick, and J. Malik. Hypercolumns for object segmentation and fine-grained localization. CoRR, abs/1411.5752, 2014. 3 [13] P.-J. Hsieh, E. Vul, and N. Kanwisher. Recognition alters the spatial pattern of fmri activation in early retinotopic cortex. Journal of Neurophysiology, 103(3):1501–1507, 2010. 1, 8 [14] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp boundary detection using pointwise mutual information. In ECCV, 2014. 2, 4 [15] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 4 [16] J. J. Kivinen, C. K. Williams, N. Heess, and D. Technologies. Visual boundary prediction: A deep neural prediction network and quality dissection. AISTATS, 1(2):9, 2014. 2, 3, 4, 5 [17] Z. Kourtzi and N. Kanwisher. Representation of perceived object shape by the human lateral occipital complex. Science, 293:1506–1509, 2001. 1, 8 [18] J. Lim, C. L. Zitnick, and P. Dollár. Sketch tokens: A learned mid-level representation for contour and object detection. In CVPR, 2013. 2 [19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. CVPR (to appear), Nov. 2015. 3, 5 [20] M. Maire, S. X. Yu, and P. Perona. Reconstructive sparse code transfer for contour detection and semantic labeling. In Asian Conference on Computer Vision (ACCV), 2014. 2 [21] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of exemplar-svms for object detection and beyond. In ICCV, 2011. 4 [22] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision, volume 2, pages 416–423, July 2001. 4 [23] X. Ren and L. Bo. Discriminatively Trained Sparse Code Gradients for Contour Detection. In Advances in Neural Information Processing Systems, December 2012. 2, 4 [24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, 2014. 3 [25] J. L. Sanguinetti, J. J. Allen, and M. A. Peterson. The ground side of an object perceived as shapeless yet processed for semantics. Psychological science, page 0956797613502814, 2013. 1, 8 [26] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR, abs/1312.6229, 2013. 3 [27] W. Shen, X. Wang, Y. Wang, X. Bai, and Z. Zhang. Deepcontour: A deep convolutional feature learned by positivesharing loss for contour detection. June 2015. 1, 3, 4, 5 [28] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22:888–905, 1997. 2, 6 [29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 1 [30] A. Sironi, V. Lepetit, and P. Fua. Multiscale centerline detection by learning a scale-space distance transform. June 2014. 4 [31] C. L. Zitnick and P. Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014. 1, 8 Dense Optical Flow Prediction from a Static Image Jacob Walker, Abhinav Gupta, and Martial Hebert Robotics Institute, Carnegie Mellon University arXiv:1505.00295v1 [cs.CV] 2 May 2015 {jcwalker, abhinavg, hebert}@cs.cmu.edu Abstract Given a scene, what is going to move, and in what direction will it move? Such a question could be considered a non-semantic form of action prediction. In this work, we present predictive convolutional neural networks (P-CNN). Given a static image, P-CNN predicts the future motion of each and every pixel in the image in terms of optical flow. Our P-CNN model leverages the data in tens of thousands of realistic videos to train our model. Our method relies on absolutely no human labeling and is able to predict motion based on the context of the scene. Since P-CNNs make no assumptions about the underlying scene they can predict future optical flow on a diverse set of scenarios. In terms of quantitative performance, P-CNN outperforms all previous approaches by large margins. 1. Introduction Consider the images shown in Figure 1. Given the girl in front of the cake, we humans can easily predict that her head will move downward to extinguish the candle. The man with the discus is in a position to twist his body strongly to the right, and the squatting man on the bottom has nowhere to move but up. Humans have an amazing ability to not only recognize what is present in the image but also predict what is going to happen next. Prediction is an important component of visual understanding and cognition. In order for computers to react to their environment, simple activity detection is not always sufficient. For successful interactions, robots need to predict the future and plan accordingly. While prediction is still a relatively new problem, there has been some recent work that has focused on this task. The most common approach to this prediction problem is to use a planning-based agent-centric approach: an object [10] or a patch [24] is modeled as an agent that performs actions based on its current state and the goal state. Each action is decided based on compatibility with the environment and how this actions helps the agent move closer to the goal state. The priors on actions are modeled via transition matrices. Such an approach has been shown to produce im- (a) Input Image (b) Prediction Figure 1: Motion Prediction. Consider single, static input images (a). Our method can first identify what these actions are and predict (b) correct motion based on the pose and stage of the action without any video information. We use the color coding from [1] shown on the right. pressive results: predicting trajectories of humans in parking lots [10] or hallucinating car movements on streets [24]. There are two main problems with this approach. First, the predictions are still sparse, and the motion is still modeled as a trajectory. Second, and more importantly, these approaches have always been shown to perform in restrictive domains such as parking lots or streets. 1 In this paper, we take the next step towards generalized prediction — a framework that can be learned from tens of thousands of realistic videos. The framework can work in indoor and outdoor environments if the agent is an animal, a human, or even a car. It can account for one or multiple agents. Specifically, we look at the task of motion prediction — given a static image we predict the dense expected optical flow as if this image was part of a video. This optical flow represents how and where each and every pixel in the image is going to move in the future. Of course, we can see that motion prediction is a highly-context dependent problem. The future motion not only depends on what is active in the scene but also its context. For example, someone’s entire body may move up or down if they are jumproping, but most of the body will be stationary if they are playing the flute. Instead of modeling agents and its context separately under restrictive assumptions, we use a learning based approach for motion prediction. Specifically, we train a deep network that can incorporate all of this contextual information to make accurate predictions of future motion in a wide variety of scenes. We train our model from thousands of realistic video datasets, namely the UCF-101 [21] and the HMDB-51 [13]. Contributions: Our paper makes threefold contributions. First, we present a predictive-CNN (P-CNN) model for motion prediction. Given a static image, our P-CNN model predicts expected motion in terms of optical flow. Our CNN-based model is agent-free and makes almost no assumptions about the underlying scene. Therefore, we show experimental results on diverse set of scenes. Second, PCNN gives state of the art performance on prediction compared to contemporary approaches. Finally, we also present an proof of concept extension of P-CNN which makes longrange prediction about future motion. Our preliminary results indicate the P-CNN might indeed be promising even on the task of long-range prediction. 2. Background Prediction has caught the interest of the vision community in recent years. Most of research in this area has looked at different aspects of the problem. The first aspect of interest is what should be predicted: what is the output space of prediction. Some of the initial work in this area focused on predicting the optical flow for the given input image [28]. Others have looked at more semantic forms of prediction: that is, predict the action class of what is going to happen next [5, 14]. But one of the issues with semantic prediction is that it tells us nothing about the future action beyond the category. One of our goals in prediction is to go beyond classification and predict the spatial layout of future actions. For example, in case of agents such as humans, the output space of prediction can be trajectories themselves [10]. But recent approaches have argued for much richer form of pre- dictions even in terms of pixels [24] or the features of the next frame [7, 19]. The other aspect of research in visual prediction looks at the question: what is the right approach for prediction? There have been two classes of approaches for the temporal prediction. The first is data-driven, non-parametric approach. In case of non-parameteric approaches, they do not make any assumptions about the underlying scene. For example, [28] simply retrieves videos visually similar to the static scene, allowing a warping [16] of the matched action into the scene. The other end of the spectrum is parametric and domain-specific approaches. Here, we make assumptions of what are the active elements in the scene whether they may be cars or people. Once the assumption is made, then a model is developed to predict agent behavior. This includes forecasting pedestrian trajectories [10], human-human interactions [7], [14], human expressions through SOSVM [5], and human-object interaction through graphical models [11], [3]. Some of the recent work in this area has looked at a more hybrid approach. For example, Walker et al. [24] builds a data-derived dictionary of rigid objects given a video domain and then makes long-term motion and appearance predictions using a transition and context model. Recent approaches such as [19] and [22] have even looked at training convolutional neural networks for predicting one future frame in a clip [19] or motion of handwritten characters [22]. We make multiple advances over previous work in this paper. First, our unsupervised method can generalize across a large number of diverse domains. While [24] does not explicitly require video labels, it is still domain dependent, requiring a human-given distinction between videos in and outside the domain. In addition, [24] focused only on birdseye domains where scene depth was limited or non existent, while our method is able to generalize to scenes with perspective. [18] also uses unsupervised methods to train a Structured Random Forest for motion prediction. However, the authors only learn a model from the simple KTH [15] dataset. We show that our method is able to learn from a set of videos that is far more diverse across scenes and actions. In addition, we demonstrate much better generalization can be obtained as compared to the nearest-neighbor approach of Yuen et al. [28]. Convolutional Neural Networks: We show in this paper that a convolutional neural network can be trained for the task of motion prediction in terms of optical flow. Current work on CNNs have largely focused on recognition tasks both in images and video [12, 9, 4, 23, 29, 20, 22]. There has been some initial work where CNNs have been combined with recurrent models for prediction. For example, [19] uses a LSTM [6] to predict the immediate next frame given a video input. [22] uses a recurrent architecture to (a) Input Image (b) Prediction (c) Ground Truth Figure 3: Consider the images on the left. Is the man squatting up or down? The bottom is near completion (or just starting), and the top image is right in the middle of the action. Our dataset contains a large number of ambiguous images such as these. In our evaluation we consider the underlying distribution of movements predicted by our network. It is highly likely that this man is going to move up or down, but unlikely that he will veer off to the left or right. predict motions of handwritten characters from a video. On the other hand, our approach predicts motion for each and every pixel from a static image for any generic scene. 3. Methods Our goal is to learn a mapping between the input RGB image and the output space which corresponds to the predicted motion of each and every pixel in terms of optical flow. We propose to use CNNs as the underlying learning algorithm for this task. However, there are few questions that need to be answered: what is a good output space and what is a good loss function? Should we model optical flow prediction as a regression or a classification problem? What is a good architecture to solve this problem? We now discuss these issues below. 3.1. Regression as Classification Intuitively, motion estimation can be posed as a regression problem since the space is continuous. Indeed, this is exactly the approach used in [18], where the authors used structured random forests to regress the magnitude and direction of the optical flow. However, such an approach has one drawback: such an output space tends to smoothen results as the ambiguity is handled by averaging out the flow. Interestingly, in a related problem of surface normal prediction, researchers have proposed reformulating structured regression as a classification problem [26, 17]. Specifically, they quantize the surface normal vectors into a codebook of clusters and then output space becomes predicting the cluster membership. In our work, we take a similar approach. We quantize optical flow vectors into 40 clusters by k-means. We can then treat the problem in a manner similar to semantic segmentation, where we classify each region as the image as a particular cluster of optical flow. We use a soft-max loss layer at the output for computing gradients. However, at test time, we create a soft output by considering the underlying distribution of all the clusters, taking a weighted-probability sum over all the classes in a given pixel for the final output. Transforming the problem into classification also leads directly to a discrete probability distribution over vector directions and magnitudes. As the problem of motion prediction can be ambiguous depending on the image (see Figure 3), we can utilize this probability distribution over directions to measure how informative our predictions are. We may be unsure if the man in Figure 3 is sitting down or standing up given only the image, but we can be quite sure he will not turn right or left. In the same way, our network can rank upward and downward facing clusters much higher than other directions. Even if the ground truth is upward, and the highest ranked cluster is downward, it may be that the second-highest cluster is also upward. A discrete probability distribution, through classification, allows an easier understanding of how well our network may be performing. In addition, we can simply compute the entropy of the distribution, allowing us to compute the confidence of our motion prediction and retrieve images that are more likely to be correct. 3.2. Network Design Our model is similar to the standard seven-layer architecture from [12]. To simplify the description, we denote the convolutional layer as C(k, s), which indicates the there are k kernels, each having the size of s × s. During convolution, we set all the strides to 1 except for the first layer, which is 4. We also denote the local response normalization layer as LRN, and the max-pooling layer as MP. The stride for pooling is 2 and we set the pooling operator size as 3 × 3. Finally, F (n) denotes fully connected layer with n neurons. Our network architecture can be described as: This can be described as: C(96, 11) → LRN → P → C(256, 5) → LRN → P → C(384, 3) → C(384, 3) → C(256, 3) → P → F (4096) → F (4096). We used a modified version of the popular Caffe toolbox [8] for our implementation. For computational simplicity, we use 200x200 windows as input. We used a learning rate of 0.0001 and a stepsize of 50000 iterations. Other network parameters were set to default. The only exception is that we used Xavier initialization of parameters. Instead of using the default softmax output, we used a spatial softmax loss function from [26] to classify every region in the image. This leads to a M ×N ×K softmax layer, where M is the number of rows, N is the number of columns, and C is the number of clusters in our codebook. We used M = 20, N = 20, and K = 40 for a softmax layer of 16,000 neurons. Our Figure 2: Overview. Our network is similar to the standard 7-layer architecture [12] used for many recognition tasks. We take a 200x200 image as input. However, we use a spatial softmax as the final output. For every pixel in the image we predict a distribution of various motions with various directions and magnitudes. We can combine a weighted average of these vectors to produce the final output for each pixel. For computational reasons, we predict a coarse 20x20 output. softmax loss is spatial, summing over all the individual region losses. Let I represent the image and Y be the ground truth optical flow labels represented as quantized clusters. Then our spatial loss function L(I, Y ) is following: L(I, Y ) = − K M ×N X X (1(yi = r) log Fi,r (I)) (1) i=1 r=1 Fi,r (I) represents the probability that the ith pixel will move according to cluster r. 1(yi = r) is an indicator function. Data Augmentation: For many deep networks, datasets which are insufficiently diverse or too small will lead directly to overfitting. [20] and [9] show that training directly on datasets such as the UCF-101 for action classification leads to overfitting, as there are only data on the order of tens of thousands of videos. However, our problem of single-frame prediction is different from this task. We find that we are able to build a generalizable representation for prediction by training our model over 350,000 frames from the UCF-101 dataset as well as over 150,000 frames from the HMDB-51 dataset. We benefit additionally from data augmentation techniques. We randomly flip each image as well as use randomly cropped windows. For each input, we also change our labels according to our spatial transformation. In this way we are able to avoid spatial biases (such as humans always appearing in the middle of the image) and train a general model on a far smaller set of videos than for recognition tasks. Labeling: We automatically label our training dataset with an optical flow algorithm. With a publically available implementation, we chose DeepFlow [27] to compute optical flow. The UCF-101 and the HMDB-51 dataset use realistic, sometimes low-quality videos from a wide variety of sources. They often suffer from compression artifacts. Thus, we aim to make our labels somewhat less noisy by taking the average optical flow of five future frames for each image. The videos in these datasets are also unstabilized. [25] showed that action recognition can be greatly improved with camera stabilization. In order to further denoise our labels, we wish to focus on the motion of objects inside the image, not the camera motion. We thus use the stabilization portion of the implementation of [25] to automatically stabilize videos using an estimated homography. 4. Experiments For our experiments, we focused on two datasets, the UCF-101 and HMDB-51, which have been popular for action recognition. For both of these datasets, we compared against baselines using 3-fold cross validation with the splits specified by the dataset organizers. For training, we subsampled frames by a factor of 5. For testing, we sampled 26,000 frames per split. For our comparison with AlexNet finetuning, we used a split which incorporated a larger portion of the training data. We will release this split publicly. We used three baselines for evaluation. First we used the technique of [18], a SRF approach to motion prediction. We took their publicly available implementation and trained a model according to their default parameters. Because of the much larger size of our datasets, we had to sample SIFT-patches less densely. We also use a NearestNeighbor baseline using both fc7 features from the pretrained AlexNet network as well as pooled-5 features. Finally, we compare unsupervised training from scratch with finetuning on the supervised AlexNet network. (a) Input Image (b) Prediction (c) Ground Truth (a) Input Image (b) Prediction (c) Ground Truth Figure 4: Qualitative results from our method for the single frame model. Our network can find the active elements in the scene and correctly predict future motion based on the context in a wide variety and scenes and actions. The color coding is on the right. 4.1. Evaluation Metrics Because of the complexity and sometimes high level of label ambiguity in motion prediction, we use a variety of metrics to evaluate our method and baselines. Following from [18], we use traditional End-Point-Error, measuring the Euclidean distance of the estimated optical flow vector from the ground truth vector. In addition, given vectors x1 and x2 , we also measure direction similarity using the coxT x sine similarity distance: kx11kkx22 k and orientation similarity |xT x | (angle taken on half-circle): kx11kkx22 k The orientation similarity measures how parallel is predicted optical flow vector with respect to given ground truth optical flow vector. Some motions may be strictly left-right or up-down, but the exact direction may be ambiguous. This measure accounts for this situation. We choose these metrics established by earlier work. However, we also add some additional metrics to account for the level of ambiguity in many of the test images. As [18] notes, EPE is a poor metric in the case where motion is small and may reasonably proceed in more than one possible direction. We thus additionally look at the underlying distribution of the predicted classes to understand how well the algorithm accounts for this ambiguity. For instance, if we are shown an image as in Figure 3, it is unknown if the man will move up or down. It is certainly the case, however, that he will not move right or left. Given the probability distribution over the quantized flow clusters, we check to see if the ground truth is within the top probable clusters. For the implementation of [18], we create an estimated probability distribution by quantizing the regression output from all the trees and then, for each pixel, we bin count the clus- ImageNet-Pretrained vs. Trained from Scratch Method EPE EPE-Canny EPE-NZ Pretrained 1.19 1.12 3.12 From Scratch 1.28 1.21 3.21 — Orient Orient-Canny Orient-NZ Pretrained .661 .659 .692 From Scratch .659 .658 .691 — Top-5 Top-5-Canny Pretrained 91.0% 91.1% From Scratch 89.9% 90.3% Top-5-NZ 65.8% 65.1% Table 1: Here we compare our unsupervised network, trained from scratch, to the same network fine-tuned from supervised AlexNet features on UCF-101. The Canny suffix represents pixels on the Canny edges, and the NZ suffix represents moving pixels according to the ground-truth. NN represents a nearest-neighbor approach. Dir and Orient represent direction and orientation metrics respectively. For EPE, less is better, and for other metrics, higher is better. ters over the trees. For Nearest-Neighbor we take the top-N matched frames and use the matched clusters in each pixel as our top-N ranking. We evaluate over the mean rank of all pixels in the image. Following [18], we also evaluate over the Canny edges. Because of the simplicity of the datasets in [18], Canny edges were a good approximation for measuring the error of pixels of moving objects in the scene. However, our data includes highly cluttered scenes that incorporate multiple non-moving objects. In addition, we find that our network is very effective at identifying moving vs non-moving elements in the scene. We find that the difference between overall pixel mean and Canny edges is very small across all metrics and baselines. Thus, we also evaluate over the moving pixels according to the ground-truth. Moving pixels in this case includes all clusters in our codebook except for the vector of smallest magnitude. While unfortunately this metric depends on the choice of codebook, we find that the greatest variation in performance and ambiguity lies in predicting the direction and magnitude of the active elements in the scene. 4.2. Qualitative Results Figure 4 shows some of our qualitative results. For single frame prediction, our network is able to predict motion in many different contexts. Although most of our datasets consist of human actions, our model can generalize beyond simply detecting general motion on humans. Our method is able to successfully predict the falling of the ocean wave in the second row, and it predicts the motion of the entire horse in the first row. Furthermore, our network can specify motion depending on the action being performed. For the man playing guitar and the man writing on the wall, the arm is Method SRF [18] NN pooled-5 NN fc7 Ours UCF-101 EPE EPE-Canny 1.30 1.23 2.31 2.20 2.24 2.16 1.27 1.17 EPE-NZ 3.24 4.40 4.27 3.19 — SRF [18] NN pooled-5 NN fc7 Ours Dir .004 -.001 -.005 .045 Dir-NZ -.013 -.067 -.060 .092 — SRF [18] NN pooled-5 NN fc7 Ours Orient Orient-Canny Orient-NZ .492 .600 .515 .650 .650 .677 .649 .649 .651 .659 .657 .688 — SRF [18] NN pooled-5 NN fc7 Ours Top-5 Top-5-Canny Top-5-NZ 79.4% 81.7% 10.0% 77.8% 79.5% 20.0% 78.3% 79.9% 18.8% 89.7% 90.5% 65.0% Dir-Canny .000 -.001 -.006 .025 — Top-10 Top-10-Canny Top-10-NZ SRF [18] 82.2% 84.4% 17.2% NN pooled-5 83.2% 85.3% 32.9% NN fc7 84.0% 85.4% 32.3% Ours 96.5% 96.7% 90.9% Table 2: Single-image evaluation using the 3-fold split on UCF101. The Canny suffix represents pixels on the Canny edges, and the NZ suffix represents moving pixels according to the groundtruth. NN represents a nearest-neighbor approach. Dir and Orient represent direction and orientation metrics respectively. For EPE, less is better, and for other metrics, higher is better. the most salient part to be moved. For the man walking the dog and the man doing a pushup, the entire body will move according to the action. 4.3. Quantitative Results We show in tables 2 and 3 that our method strongly outperforms both the Nearest-Neighbor and SRF-based baselines by a large margin by most metrics. This holds true for both datasets. Interestingly, the SRF-based approach seems to come close to ours based on End-Point-Error, but is heavily outperformed on all other metrics. This is largely a product of the End-Point-Error metric, as we find that the SRF tends to output the mean (optical flow with very small magnitude). This is consistent with the results found in [18], where actions with low, bidirectional motion can result in higher EPE than predicting no motion at all. When we account for this ambiguity in motion in the top-N metric, however, the difference in performance is large. Compared to unsupervised training from scratch, we find that finetuning Method SRF [18] NN pooled-5 NN fc7 Ours HMDB-51 EPE EPE-Canny 1.23 1.20 2.51 2.49 2.43 2.43 1.21 1.17 EPE-NZ 3.46 4.89 4.69 3.45 — SRF [18] NN pooled-5 NN fc7 Ours Dir .000 -.008 -.007 .019 Dir-NZ -.010 -.061 -.061 .030 — SRF [18] NN pooled-5 NN fc7 Ours Orient Orient-Canny Orient-NZ .461 .557 .495 .631 .631 .644 .630 .631 .655 .636 .636 .667 — SRF [18] NN pooled-5 NN fc7 Ours Top-5 Top-5-Canny Top-5-NZ 81.9% 83.6% 13.5% 76.3% 77.8% 14.0% 77.3% 78.7% 13.5% 90.2% 90.5% 61.0% Dir-Canny .000 -.007 -.005 .012 — Top-10 Top-10-Canny Top-10-NZ SRF [18] 84.4% 86.1% 22.1% NN pooled-5 82.9% 84.0% 23.9% NN fc7 83.6% 84.4% 23.2% Ours 95.9% 95.9% 87.5% Table 3: Single-image evaluation using the 3-fold split on HMDB-51. The Canny suffix represents pixels on the Canny edges, and the NZ suffix represents moving pixels according to the ground-truth. NN represents a nearest-neighbor approach. Dir and Orient represent direction and orientation metrics respectively. For EPE, less is better, and for other metrics, higher is better. from supervised, pretrained features yield only a very small improvement in performance. Looking over all pixels, the difference in performance between approaches is small. On absolute levels, the orientation and top-N metrics also tend to be high. This is due to the fact that most pixels in the image are not going to move. Outputting low or zero-motion over the entire image can thus lead to good performance for many metrics. Canny edge pixels yield similar results, as our natural images often include background clutter with objects that do not move. The most dramatic differences appear over the non-zero pixels. The direction metric is for our method is low at .09 because of direction ambiguity, but orientation similarity is much larger. The largest performance gains come at the top-N rankings. For 40 clusters, random chance for top-5 is 12.5%, and for top-10 it is 25%. Nearest-neighbor does slightly better than chance, but SRF actually performs slightly worse. This is most likely due to the SRF tendency to output the overall mean flow, which is of low magnitude. Our method performs much better, with the ground truth direction and magnitude vector coming in 65% of the time in the top-5 ranking, and to a very high 90.9% of the time in the top ten. In spite of exposure to more data and use of semantic labels, we see in Table 1 that finetuning using the ImageNet network only leads to a small increase in performance compared to training from scratch. 5. Multi-Frame Prediction Until now we have described an architecture for predicting optical flow given a static image as input. However, it would be interesting to predict not just the next frame but a few seconds into future. How should we design such a network? We present a proof-of-concept network to predict 6 future frames. In order to predict multiple frames into the future, we take our pre-trained single frame network and output the seventh feature layer into a ”temporally deep” network, using the implementation of [2]. This network architecture is the same as an unrolled recurrent neural network with some important differences. On a high level, our network is similar to the unfactored architecture in [2], with each sequence having access to the image features and the previous hidden state in order to predict the next state. We replace the LSTM module with a fully connected layer as in a RNN. However, we also do not use a true recurrent network. The weights for each sequence layer are not shared, and each sequence has access to all the past hidden states. We used 2000 hidden states in our network, but we predict at most six future sequences. We attempted to use recurrent architectures with the publicly available LSTM implementation from [2]. However, in our experiments they always regressed to a mean trajectory across the data. Our fully connected network has much higher number of parameters than a RNN and therefore highlights the inherent difficulty of this task. Due to the much larger size of the state space, we do not predict optical flow for each and every pixel. Instead, we use kmeans to created a codebook of 1000 possible optical flow frames, and we predict one of 1000 class as output as each time step. This can be thought of as analogous to a sequential prediction problem similar to caption generation. Instead of a sequence of words, our ”words” are clusters of optical flow frames, and our ”sentence” is an entire trajectory. We used a set number of sequences, six, in our experiments with each frame representing the average optical flow of one-sixth of a second. 6. Conclusion In this paper we have presented an approach to generalized event prediction in static scenes. Namely, our framework focuses on motion prediction as a non-semantic form of action prediction. By using an optical flow algorithm Figure 5: Overview. For our multiframe prediction, we predict entire clustered frames of optical flow as a sequence of frames. We take the learned features for our single frame model as our input, and we input them to a series of six fully connected layers, with each layer having access to the states of the past layers. (a) Input Image (b) Frame 1 (c) Frame 2 (c) Frame 3 (d) Frame 4 (e) Frame 5 Figure 6: Qualitative results for multi-frame prediction. The four rows represent predictions from our multi-frame model for future frames. Our extension can predict optical flow over multiple frames. to label the data, we can train this model on a large number of unsupervised videos. Furthermore, our framework utilizes the success of deep networks to outperform contemporary approaches to motion prediction. We find that our network successfully predicts motion based on the context of the scene and the stage of the action taking place. Possible work includes incorporating this motion model to predict semantic action labels in images and video. Another possible direction is to utilize the predicted optical flow to predict in raw pixel space, synthesizing a video from a single image. Acknowedgements: We thank Xiaolong Wang for many helpful discussions. We thank the NVIDIA Corporation for the donation of Tesla K40 GPUs for this research. In addition, this work was supported by NSF grant IIS1227495. References [1] S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. IJCV, 92(1):1–31, 2011. 1 [2] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. arXiv preprint arXiv:1411.4389, 2014. 7 [3] D. Fouhey and C. L. Zitnick. Predicting object dynamics in scenes. In CVPR, 2014. 2 [4] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 2 [5] M. Hoai and F. De la Torre. Max-margin early event detectors. IJCV, 107(2):191–202, 2014. 2 [6] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 2 [7] D.-A. Huang and K. M. Kitani. Action-reaction: Forecasting the dynamics of human interaction. In ECCV. 2014. 2 [8] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 3 [9] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014. 2, 4 [10] K. Kitani, B. Ziebart, D. Bagnell, and M. Hebert. Activity forecasting. In ECCV, 2012. 1, 2 [11] H. S. Koppula and A. Saxena. Anticipating human activities using object affordances for reactive robotic response. In RSS, 2013. 2 [12] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097–1105, 2012. 2, 3, 4 [13] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. In ICCV, 2011. 2 [14] T. Lan, T.-C. Chen, and S. Savarese. A hierarchical representation for future action prediction. In ECCV. 2014. 2 [15] I. Laptev, B. Caputo, et al. Recognizing human actions: A local svm approach. In ICPR, 2004. 2 [16] C. Liu, J. Yuen, and A. Torralba. Sift flow: Dense correspondence across scenes and its applications. PAMI, 2011. 2 [17] B. Z. Lubor Ladickỳ and M. Pollefeys. Discriminatively trained dense surface normal estimation. 3 [18] S. L. Pintea, J. C. van Gemert, and A. W. Smeulders. Déjà vu: Motion prediction in static images. In ECCV. 2014. 2, 3, 4, 5, 6, 7 [19] M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604, 2014. 2 [20] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014. 2, 4 [21] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 2 [22] N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using lstms. arXiv preprint arXiv:1502.04681, 2015. 2 [23] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. 2 [24] J. Walker, A. Gupta, and M. Hebert. Patch to the future: Unsupervised visual prediction. In CVPR, 2014. 1, 2 [25] H. Wang and C. Schmid. Action recognition with improved trajectories. In ICCV, Sydney, Australia, 2013. 4 [26] X. Wang, D. F. Fouhey, and A. Gupta. Designing deep networks for surface normal estimation. arXiv preprint arXiv:1411.4958, 2014. 3 [27] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. DeepFlow: Large displacement optical flow with deep matching. In ICCV, 2013. 4 [28] J. Yuen and A. Torralba. A data-driven approach for event prediction. In ECCV, 2010. 2 [29] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev. Panda: Pose aligned networks for deep attribute modeling. In CVPR, 2014. 2 Ask Your Neurons: A Neural-based Approach to Answering Questions about Images arXiv:1505.01121v3 [cs.CV] 1 Oct 2015 Mateusz Malinowski1 Marcus Rohrbach2 Mario Fritz1 1 Max Planck Institute for Informatics, Saarbrücken, Germany 2 UC Berkeley EECS and ICSI, Berkeley, CA, United States Abstract We address a question answering task on real-world images that is set up as a Visual Turing Test. By combining latest advances in image representation and natural language processing, we propose Neural-Image-QA, an endto-end formulation to this problem for which all parts are trained jointly. In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question). Our approach Neural-Image-QA doubles the performance of the previous best approach on this problem. We provide additional insights into the problem by analyzing how much information is contained only in the language part for which we provide a new human baseline. To study human consensus, which is related to the ambiguities inherent in this challenging task, we propose two novel metrics and collect additional answers which extends the original DAQUAR dataset to DAQUAR-Consensus. 1. Introduction With the advances of natural language processing and image understanding, more complex and demanding tasks have become within reach. Our aim is to take advantage of the most recent developments to push the state-of-theart for answering natural language questions on real-world images. This task unites inference of question intends and visual scene understanding with a word sequence prediction task. Most recently, architectures based on the idea of layered, end-to-end trainable artificial neural networks have improved the state of the art across a wide range of diverse tasks. Most prominently Convolutional Neural Networks have raised the bar on image classification tasks [16] and Long Short Term Memory Networks are dominating performance on a range of sequence prediction tasks such as machine translation [28]. Very recently these two trends of employing neural architectures have been combined fruitfully with methods that CNN What is behind the table ? LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM chairs window <END> Figure 1. Our approach Neural-Image-QA to question answering with a Recurrent Neural Network using Long Short Term Memory (LSTM). To answer a question about an image, we feed in both, the image (CNN features) and the question (green boxes) into the LSTM. After the (variable length) question is encoded, we generate the answers (multiple words, orange boxes). During the answer generation phase the previously predicted answers are fed into the LSTM until the hENDi symbol is predicted. can generate image [12] and video descriptions [30]. Both are conditioning on the visual features that stem from deep learning architectures and employ recurrent neural network approaches to produce descriptions. To further push the boundaries and explore the limits of deep learning architectures, we propose an architecture for answering questions about images. In contrast to prior work, this task needs conditioning on language as well visual input. Both modalities have to be interpreted and jointly represented as an answer depends on inferred meaning of the question and image content. While there is a rich body of work on natural language understanding that has addressed textual question answering tasks based on semantic parsing, symbolic representation and deduction systems, which also has seen applications to question answering on images [20], there is initial evidence that deep architectures can indeed achieve a similar goal [33]. This motivates our work to seek end-to-end architectures that learn to answer questions in a single holistic and monolithic model. We propose Neural-Image-QA, an approach to question answering with a recurrent neural network. An overview is given in Figure 1. The image is analyzed via a Convolutional Neural Network (CNN) and the question together with the visual representation is fed into a Long Short Term Memory (LSTM) network. The system is trained to produce the correct answer to the question on the image. CNN and LSTM are trained jointly and end-to-end starting from words and pixels. Contributions: We proposes a novel approach based on recurrent neural networks for the challenging task of answering of questions about images. It combines a CNN with a LSTM into an end-to-end architecture that predict answers conditioning on a question and an image. Our approach significantly outperforms prior work on this task – doubling the performance. We collect additional data to study human consensus on this task, propose two new metrics sensitive to these effects, and provide a new baseline, by asking humans to answer the questions without observing the image. We demonstrate a variant of our system that also answers question without accessing any visual information, which beats the human baseline. 2. Related Work As our method touches upon different areas in machine learning, computer vision and natural language processing, we have organized related work in the following way: Convolutional Neural Networks for visual recognition. We are building on the recent success of Convolutional Neural Networks (CNN) for visual recognition [16, 17, 25], that are directly learnt from the raw image data and pre-trained on large image corpora. Due to the rapid progress in this area within the last two years, a rich set of models [27, 29] is at our disposal. Recurrent Neural Networks (RNN) for sequence modeling. Recurrent Neural Networks allow Neural Networks to handle sequences of flexible length. A particular variant called Long Short Term Memory (LSTM) [9] has shown recent success on natural language tasks such as machine translation [3, 28]. Combining RNNs and CNNs for description of visual content. The task of describing visual content like still images as well as videos has been successfully addressed with a combination of the previous two ideas [5, 12, 31, 32, 37]. This is achieved by using the RNN-type model that first gets to observe the visual content and is trained to afterwards predict a sequence of words that is a description of the visual content. Our work extends this idea to question answering, where we formulate a model trained to generate an answer based on visual as well as natural language input. Grounding of natural language and visual concepts. Dealing with natural language input does involve the asso- ciation of words with meaning. This is often referred to as grounding problem - in particular if the “meaning” is associated with a sensory input. While such problems have been historically addressed by symbolic semantic parsing techniques [15, 22], there is a recent trend of machine learningbased approaches [12, 13, 14] to find the associations. Our approach follows the idea that we do not enforce or evaluate any particular representation of “meaning” on the language or image modality. We treat this as latent and leave this to the joint training approach to establish an appropriate internal representation for the question answering task. Textual question answering. Answering on purely textual questions has been studied in the NLP community [2, 18] and state of the art techniques typically employ semantic parsing to arrive at a logical form capturing the intended meaning and infer relevant answers. Only very recently, the success of the previously mentioned neural sequence models as RNNs has carried over to this task [10, 33]. More specifically [10] uses dependency-tree Recursive NN instead of LSTM, and reduce the questionanswering problem to a classification task. Moreover, according to [10] their method cannot be easily applied to vision. [33] propose different kind of network - memory networks - and it is unclear how to apply [33] to take advantage of the visual content. However, neither [10] nor [33] show an end-to-end, monolithic approaches that produce multiple words answers for question on images. Visual Turing Test. Most recently several approaches have been proposed to approach Visual Turing Test [21], i.e. answering question about visual content. For instance [8] have proposed a binary (yes/no) version of Visual Turing Test on synthetic data. In [20], we present a question answering system based on a semantic parser on a more varied set of human question-answer pairs. In contrast, in this work, our method is based on a neural architecture, which is trained end-to-end and therefore liberates the approach from any ontological commitment that would otherwise be introduced by a semantic parser. We like to note that shortly after this work, several neural-based models [24, 19, 7] have also been suggested. Also several new datasets for Visual Turing Tests have just been proposed [1, 35] that are worth further investigations. 3. Approach Answering questions on images is the problem of predicting an answer a given an image x and a question q according to a parametric probability measure: â = arg max p(a|x, q; θ) (1) a∈A where θ represent a vector of all parameters to learn and A is a set of all answers. Later we describe how we represent x, a, q, and p(·|x, q; θ) in more details. LSTM Unit Input Gate CNN x Output Gate σ ϕ + ϕ qn ai-1 vt ht-1 ht = zt ct Input Modulation Gate qn-1 σ σ Forget Gate ... LSTM LSTM ... a1 ... LSTM ct-1 ... Figure 3. LSTM unit. See Section 3, Equations (3)-(8) for details. ai Figure 2. Our approach Neural-Image-QA, see Section 3 for details. In our scenario questions can have multiple word answers and we consequently decompose  the problem to predicting a set of answer words aq,x = a1 , a2 , ..., aN (q,x) , where at are words from a finite vocabulary V 0 , and N (q, x) is the number of answer words for the given question and image. In our approach, named Neural-Image-QA, we propose to tackle the problem as follows. To predict multiple words we formulate the problem as predicting a sequence of words from the vocabulary V := V 0 ∪ {$} where the extra token $ indicates the end of the answer sequence, and points out that the question has been fully answered. We thus formulate the prediction procedure recursively: ât = arg max p(a|x, q, Ât−1 ; θ) (2) a∈V where Ât−1 = {â1 , . . . , ât−1 } is the set of previous words, with Â0 = {} at the beginning, when our approach has not given any answer so far. The approach is terminated when ât = $. We evaluate the method solely based on the predicted answer words ignoring the extra token $. To ensure uniqueness of the predicted answer words, as we want to predict the set of answer words, the prediction procedure can be be trivially changed by maximizing over V \ Ât−1 . However, in practice, our algorithm learns to not predict any previously predicted words. As shown in Figure 1 and Figure 2, we feed Neural-ImageQA with a question   as a sequence of words, i.e. q = q 1 , . . . , q n−1 , J?K , where each q t is the t-th word question and J?K := q n encodes the question mark - the end of the question. Since our problem is formulated as a variablelength input/output sequence, we model the parametric distribution p(·|x, q; θ) of Neural-Image-QA with a recurrent neural network and a softmax prediction layer. More precisely, Neural-Image-QA is a deep network built of CNN [17] and Long-Short Term Memory (LSTM) [9]. LSTM has been recently shown to be effective in learning a variablelength sequence-to-sequence mapping [5, 28]. Both question and answer words are represented with one-hot vector encoding (a binary vector with exactly one non-zero entry at the position indicating the index of the word in the vocabulary) and embedded in a lower dimensional space, using a jointly learnt latent linear embedding. In the training phase, we augment the question words sequence q with the corresponding ground truth answer words sequence a, i.e. q̂ := [q, a]. During the test time, in the prediction phase, at time step t, we augment q with previously predicted answer words â1..t := [â1 , . . . , ât−1 ], i.e. q̂ t := [q, â1..t ]. This means the question q and the previous answers are encoded implicitly in the hidden states of the LSTM, while the latent hidden representation is learnt. We encode the image x using a CNN and provide it at every time step as input to the LSTM. We set the input v t as a concatenation of [x, q̂ t ]. As visualized in detail in Figure 3, the LSTM unit takes an input vector v t at each time step t and predicts an output word zt which is equal to its latent hidden state ht . As discussed above zt is a linear embedding of the corresponding answer word at . In contrast to a simple RNN unit the LSTM unit additionally maintains a memory cell c. This allows to learn long-term dynamics more easily and significantly reduces the vanishing and exploding gradients problem [9]. More precisely, we use the LSTM unit as described in [36] and the Caffe implementation from [5]. With the −1 sigmoid nonlinearity σ : R 7→ [0, 1], σ(v) = (1 + e−v ) and the hyperbolic tangent nonlinearity φ : R 7→ [−1, 1], v −v φ(v) = eev −e +e−v = 2σ(2v) − 1, the LSTM updates for time step t given inputs v t , ht−1 , and the memory cell ct−1 as follows: it = σ(Wvi v t + Whi ht−1 + bi ) (3) f t = σ(Wvf v t + Whf ht−1 + bf ) (4) ot = σ(Wvo v t + Who ht−1 + bo ) (5) g t = φ(Wvg v t + Whg ht−1 + bg ) (6) ct = f t ct−1 + it (7) ht = ot φ(ct ) gt (8) where denotes element-wise multiplication. All the weights W and biases b of the network are learnt jointly with the cross-entropy loss. Conceptually, as shown in Figure 3, Equation 3 corresponds to the input gate, Equation 6 the input modulation gate, and Equation 4 the forget gate, which determines how much to keep from the previous memory ct−1 state. As Figures 1 and 2 suggest, all the output predictions that occur before the question mark are excluded from the loss computation, so that the model is penalized solely based on the predicted answer words. Implementation We use default hyper-parameters of LSTM [5] and CNN [11]. All CNN models are first pretrained on the ImageNet dataset [25], and next we randomly initialize and train the last layer together with the LSTM network on the task. We find this step crucial in obtaining good results. We have explored the use of a 2 layered LSTM model, but have consistently obtained worse performance. In a pilot study, we have found that GoogleNet architecture [11, 29] consistently outperforms the AlexNet architecture [11, 16] as a CNN model for our task and model. 4. Experiments In this section we benchmark our method on a task of answering questions about images. We compare different variants of our proposed model to prior work in Section 4.1. In addition, in Section 4.2, we analyze how well questions can be answered without using the image in order to gain an understanding of biases in form of prior knowledge and common sense. We provide a new human baseline for this task. In Section 4.3 we discuss ambiguities in the question answering tasks and analyze them further by introducing metrics that are sensitive to these phenomena. In particular, the WUPS score [20] is extended to a consensus metric that considers multiple human answers. Additional results are available in the supplementary material and on the project webpage 1 . Experimental protocol We evaluate our approach on the DAQUAR dataset [20] which provides 12, 468 human question answer pairs on images of indoor scenes [26] and follow the same evaluation protocol by providing results on accuracy and the WUPS score at {0.9, 0.0}. We run experiments for the full dataset as well as their proposed reduced set that restricts the output space to only 37 object categories and uses 25 test images. In addition, we also evaluate the methods on different subsets of DAQUAR where only 1, 2, 3 or 4 word answers are present. WUPS scores We base our experiments as well as the consensus metrics on WUPS scores [20]. The metric is a generalization of the accuracy measure that accounts for word-level ambiguities in the answer words. For instance ‘carton’ and ‘box’ can be associated with a similar concept, 1 https://www.d2.mpi-inf.mpg.de/ visual-turing-challenge Malinowski et al. [20] Neural-Image-QA (ours) - multiple words - single word Human answers [20] Language only (ours) - multiple words - single word Human answers, no images Accuracy WUPS @0.9 WUPS @0.0 7.86 11.86 38.79 17.49 19.43 50.20 23.28 25.28 50.82 57.76 62.00 67.27 17.06 17.15 7.34 22.30 22.80 13.17 56.53 58.42 35.56 Table 1. Results on DAQUAR, all classes, single reference, in %. and hence models should not be strongly penalized for this type of mistakes. Formally: WUPS(A, T ) = N Y 1 X min{ maxi µ(a, t), N i=1 t∈T a∈Ai Y maxi µ(a, t)} t∈T i a∈A To embrace the aforementioned ambiguities, [20] suggest using a thresholded taxonomy-based Wu-Palmer similarity [34] for µ. The smaller the threshold the more forgiving metric. As in [20], we report WUPS at two extremes, 0.0 and 0.9. 4.1. Evaluation of Neural-Image-QA We start with the evaluation of our Neural-Image-QA on the full DAQUAR dataset in order to study different variants and training conditions. Afterwards we evaluate on the reduced DAQUAR for additional points of comparison to prior work. Results on full DAQUAR Table 1 shows the results of our Neural-Image-QA method on the full set (“multiple words”) with 653 images and 5673 question-answer pairs available at test time. In addition, we evaluate a variant that is trained to predict only a single word (“single word”) as well as a variant that does not use visual features (“Language only”). In comparison to the prior work [20] (shown in the first row in Table 1), we observe strong improvements of over 9% points in accuracy and over 11% in the WUPS scores [second row in Table 1 that corresponds to “multiple words”]. Note that, we achieve this improvement despite the fact that the only published number available for the comparison on the full set uses ground truth object annotations [20] – which puts our method at a disadvantage. Further improvements are observed when we train only on a single word answer, which doubles the accuracy obtained Accuracy WUPS @0.9 WUPS @0.0 Neural-Image-QA (ours) 21.67 27.99 65.11 Malinowski et al. [20] Language only (ours) 19.13 25.16 61.51 Neural-Image-QA (ours) - multiple words - single word Table 2. Results of the single word model on the one-word answers subset of DAQUAR, all classes, single reference, in %. 30 WUPS 0.9 Accuracy 30 20 10 0 WUPS @0.9 WUPS @0.0 12.73 18.10 51.47 29.27 34.68 36.50 40.76 79.47 79.54 32.32 31.65 38.39 38.35 80.05 80.08 Table 3. Results on reduced DAQUAR, single reference, with a reduced set of 37 object classes and 25 test images with 297 question-answer pairs, in % 20 10 0 1 2 3 4 Words number Language only (ours) - multiple words - single word Accuracy 1 2 3 4 Words number Figure 4. Language only (blue bar) and Neural-Image-QA (red bar) “multi word” models evaluated on different subsets of DAQUAR. We consider 1, 2, 3, 4 word subsets. The blue and red horizontal lines represent “single word” variants evaluated on the answers with exactly 1 word. in prior work. We attribute this to a joint training of the language and visual representations and the dataset bias, where about 90% of the answers contain only a single word. We further analyze this effect in Figure 4, where we show performance of our approach (“multiple words”) in dependence on the number of words in the answer (truncated at 4 words due to the diminishing performance). The performance of the “single word” variants on the one-word subset are shown as horizontal lines. Although accuracy drops rapidly for longer answers, our model is capable of producing a significant number of correct two words answers. The “single word” variants have an edge on the single answers and benefit from the dataset bias towards these type of answers. Quantitative results of the “single word” model on the one-word answers subset of DAQUAR are shown in Table 2. While we have made substantial progress compared to prior work, there is still a 30% points margin to human accuracy and 25 in WUPS score [“Human answers” in Table 1]. Results on reduced DAQUAR In order to provide performance numbers that are comparable to the proposed MultiWorld approach in [20], we also run our method on the reduced set with 37 object classes and only 25 images with 297 question-answer pairs at test time. Table 3 shows that Neural-Image-QA also improves on the reduced DAQUAR set, achieving 34.68% Accuracy and 40.76% WUPS at 0.9 substantially outperforming [20] by 21.95% Accuracy and 22.6 WUPS. Similarly to previous experiments, we achieve the best performance using the “single word” variant. 4.2. Answering questions without looking at images In order to study how much information is already contained in questions, we train a version of our model that ignores the visual input. The results are shown in Table 1 and Table 3 under “Language only (ours)”. The best “Language only” models with 17.15% and 32.32% compare very well in terms of accuracy to the best models that include vision. The latter achieve 19.43% and 34.68% on the full and reduced set respectively. In order to further analyze this finding, we have collected a new human baseline “Human answer, no image”, where we have asked participants to answer on the DAQUAR questions without looking at the images. It turns out that humans can guess the correct answer in 7.86% of the cases by exploiting prior knowledge and common sense. Interestingly, our best “language only” model outperforms the human baseline by over 9%. A substantial number of answers are plausible and resemble a form of common sense knowledge employed by humans to infer answers without having seen the image. 4.3. Human Consensus We observe that in many cases there is an inter human agreement in the answers for a given image and question and this is also reflected by the human baseline performance on the question answering task of 50.20% [“Human answers” in Table 1]. We study and analyze this effect further by extending our dataset to multiple human reference answers in Section 4.3.1, and proposing a new measure – inspired by the work in psychology [4, 6, 23] – that handles disagreement in Section 4.3.2, as well as conducting additional experiments in Section 4.3.3. Fraction of data Fraction of data 100 50 0 100 Subset: No agreement Language only (ours) - multiple words - single word 50 0 0 50 100 Human agreement 0 50 100 Human agreement Figure 5. Study of inter human agreement. At x-axis: no consensus (0%), at least half consensus (50%), full consensus (100%). Results in %. Left: consensus on the whole data, right: consensus on the test data. 4.3.1 DAQUAR-Consensus In order to study the effects of consensus in the question answering task, we have asked multiple participants to answer the same question of the DAQUAR dataset given the respective image. We follow the same scheme as in the original data collection effort, where the answer is a set of words or numbers. We do not impose any further restrictions on the answers. This extends the original data [20] to an average of 5 test answers per image and question. We refer to this dataset as DAQUAR-Consensus. 4.3.2 Consensus Measures While we have to acknowledge inherent ambiguities in our task, we seek a metric that prefers an answer that is commonly seen as preferred. We make two proposals: Average Consensus: We use our new annotation set that contains multiple answers per question in order to compute an expected score in the evaluation: N K Y Y 1 XX min{ maxi µ(a, t), max µ(a, t)} N K i=1 a∈Ai t∈Tk i i k=1 a∈A t∈Tk Neural-Image-QA (ours) - multiple words - single word Subset: ≥ 50% agreement Language only (ours) - multiple words - single word Neural-Image-QA (ours) - multiple words - single word Subset: Full Agreement Language only (ours) - multiple words - single word Neural-Image-QA (ours) - multiple words - single word Min Consensus: The Average Consensus Metric puts more weights on more “mainstream” answers due to the summation over possible answers given by humans. In order to measure if the result was at least with one human in WUPS @0.9 WUPS @0.0 8.86 8.50 12.46 12.05 38.89 40.94 10.31 9.13 13.39 13.06 40.05 43.48 21.17 20.73 27.43 27.38 66.68 67.69 20.45 24.10 27.71 30.94 67.30 71.95 27.86 25.26 35.26 32.89 78.83 79.08 22.85 29.62 33.29 37.71 78.56 82.31 Table 4. Results on DAQUAR, all classes, single reference in % (the subsets are chosen based on DAQUAR-Consensus). agreement, we propose a “Min Consensus Metric (MCM)” by replacing the averaging in Equation 9 with a max operator. We call such metric Min Consensus and suggest using both metrics in the benchmarks. We will make the implementation of both metrics publicly available.   N Y Y 1 X K  max min{ maxi µ(a, t), max µ(a, t)} N i=1 k=1 a∈Ai t∈Tk i i a∈A t∈Tk (10) (9) where for the i-th question Ai is the answer generated by the architecture and Tki is the k-th possible human answer corresponding to the k-th interpretation of the question. Both answers Ai and Tki are sets of the words, and µ is a membership measure, for instance WUP [34]. We call this metric “Average Consensus Metric (ACM)” since, in the limits, as K approaches the total number of humans, we truly measure the inter human agreement of every question. Accuracy Intuitively, the max operator uses in evaluation a human answer that is the closest to the predicted one – which represents a minimal form of consensus. 4.3.3 Consensus results Using the multiple reference answers in DAQUARConsensus we can show a more detailed analysis of inter human agreement. Figure 5 shows the fraction of the data where the answers agree between all available questions (“100”), at least 50% of the available questions and do not agree at all (no agreement - “0”). We observe that for the majority of the data, there is a partial agreement, but even full disagreement is possible. We split the dataset Accuracy Average Consensus Metric Language only (ours) - multiple words - single word Neural-Image-QA (ours) - multiple words - single word Min Consensus Metric Language only (ours) - multiple words - single word Neural-Image-QA (ours) - multiple words - single word WUPS @0.9 Accuracy WUPS @0.9 WUPS @0.0 WUPS [20] 50.20 50.82 67.27 ACM (ours) MCM (ours) 36.78 60.50 45.68 69.65 64.10 82.40 WUPS @0.0 11.60 11.57 18.24 18.97 52.68 54.39 11.31 13.51 18.62 21.36 53.21 58.03 Table 6. Min and Average Consensus on human answers from DAQUAR, as reference sentence we use all answers in DAQUARConsensus which are not in DAQUAR, in % bed is either a pillow or doll. 22.14 22.56 29.43 30.93 66.88 69.82 22.74 26.53 30.54 34.87 68.17 74.51 Table 5. Results on DAQUAR-Consensus, all classes, consensus in %. into three parts according to the above criteria “No agreement”, “≥ 50% agreement” and “Full agreement” and evaluate our models on these splits (Table 4 summarizes the results). On subsets with stronger agreement, we achieve substantial gains of up to 10% and 20% points in accuracy over the full set (Table 1) and the Subset: No agreement (Table 4), respectively. These splits can be seen as curated versions of DAQUAR, which allows studies with factored out ambiguities. The aforementioned “Average Consensus Metric” generalizes the notion of the agreement, and encourages predictions of the most agreeable answers. On the other hand “Min Consensus Metric” has a desired effect of providing a more optimistic evaluation. Table 5 shows the application of both measures to our data and models. Moreover, Table 6 shows that “MCM” applied to human answers at test time captures ambiguities in interpreting questions by improving the score of the human baseline from [20] (here, as opposed to Table 5, we exclude the original human answers from the measure). It also cooperates well with WUPS at 0.9, which takes word ambiguities into account, gaining an about 20% higher score. 4.4. Qualitative results We show predicted answers of different variants of our architecture in Table 7, 8, and 9. We have chosen the examples to highlight differences between Neural-Image-QA and the “Language only”. We use a “multiple words” approach only in Table 8, otherwise the “single word” model is shown. Despite some failure cases, “Language only” makes “reasonable guesses” like predicting that the largest object could be table or an object that could be found on the 4.5. Failure cases While our method answers correctly on a large part of the challenge (e.g. ≈ 35 WUPS at 0.9 on “what color” and “how many” question subsets), spatial relations (≈ 21 WUPS at 0.9) which account for a substantial part of DAQUAR remain challenging. Other errors involve questions with small objects, negations, and shapes (below 12 WUPS at 0.9). Too few training data points for the aforementioned cases may contribute to these mistakes. Table 9 shows examples of failure cases that include (in order) strong occlusion, a possible answer not captured by our ground truth answers, and unusual instances (red toaster). 5. Conclusions We have presented a neural architecture for answering natural language questions about images that contrasts with prior efforts based on semantic parsing and outperforms prior work by doubling performance on this challenging task. A variant of our model that does not use the image to answer the question performs only slightly worse and even outperforms a new human baseline that we have collected under the same condition. We conclude that our model has learnt biases and patterns that can be seen as forms of common sense and prior knowledge that humans use to accomplish this task. We observe that indoor scene statistics, spatial reasoning, and small objects are not well captured by the global CNN representation, but the true limitations of this representation can only be explored on larger datasets. We extended our existing DAQUAR dataset to DAQUARConsensus, which now provides multiple reference answers which allows to study inter-human agreement and consensus on the question answer task. We propose two new metrics: “Average Consensus”, which takes into account human disagreement, and “Min Consensus” that captures disagreement in human question answering. Acknowledgements. Marcus Rohrbach was supported by a fellowship within the FITweltweit-Program of the German Academic Exchange Service (DAAD). What is on the right side of the cabinet? How many drawers are there? What is the largest object? Neural-Image-QA: bed 3 bed Language only: bed 6 table Table 7. Examples of questions and answers. Correct predictions are colored in green, incorrect in red. What is on the refrigerator? What is the colour of the comforter? What objects are found on the bed? Neural-Image-QA: magnet, paper blue, white bed sheets, pillow Language only: magnet, paper blue, green, red, yellow doll, pillow Table 8. Examples of questions and answers with multiple words. Correct predictions are colored in green, incorrect in red. How many chairs are there? What is the object fixed on the window? Which item is red in colour? Neural-Image-QA: 1 curtain remote control Language only: 4 curtain clock Ground truth answers: 2 handle toaster Table 9. Examples of questions and answers - failure cases. References [1] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. arXiv:1505.00468, 2015. [2] J. Berant and P. Liang. Semantic parsing via paraphrasing. In ACL, 2014. [3] K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, D. Bahdanau, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. [4] J. Cohen et al. A coefficient of agreement for nominal scales. Educational and psychological measurement, 1960. [5] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015. [6] J. L. Fleiss and J. Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 1973. [7] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. In NIPS, 2015. [8] D. Geman, S. Geman, N. Hallonquist, and L. Younes. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 2015. [9] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 1997. [10] M. Iyyer, J. Boyd-Graber, L. Claudino, R. Socher, and H. D. III. A neural network for factoid question answering over paragraphs. In EMNLP, 2014. [11] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093, 2014. [12] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. [13] A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. In NIPS, 2014. [14] C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to-image coreference. In CVPR, 2014. [15] J. Krishnamurthy and T. Kollar. Jointly learning to parse and perceive: Connecting natural language to the physical world. TACL, 2013. [16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. [17] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 1998. [18] P. Liang, M. I. Jordan, and D. Klein. Learning dependencybased compositional semantics. Computational Linguistics, 2013. [19] L. Ma, Z. Lu, and H. Li. Learning to answer questions from image using convolutional neural network. arXiv:1506.00333, 2015. [20] M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. [21] M. Malinowski and M. Fritz. Towards a visual turing challenge. In Learning Semantics (NIPS workshop), 2014. [22] C. Matuszek, N. Fitzgerald, L. Zettlemoyer, L. Bo, and D. Fox. A joint model of language and perception for grounded attribute learning. In ICML, 2012. [23] N. Nakashole, T. Tylenda, and G. Weikum. Fine-grained semantic typing of emerging entities. In ACL, 2013. [24] M. Ren, R. Kiros, and R. Zemel. Image question answering: A visual semantic embedding model and a new dataset. In NIPS, 2015. [25] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014. [26] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. [27] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. [28] I. Sutskever, O. Vinyals, and Q. V. V. Le. Sequence to sequence learning with neural networks. In NIPS. 2014. [29] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv:1409.4842, 2014. [30] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence – video to text. In ICCV, 2015. [31] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko. Translating videos to natural language using deep recurrent neural networks. In NAACL, 2015. [32] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. arXiv:1411.4555, 2014. [33] J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv:1410.3916, 2014. [34] Z. Wu and M. Palmer. Verbs semantics and lexical selection. In ACL, 1994. [35] L. Yu, E. Park, A. C. Berg, and T. L. Berg. Visual madlibs: Fill in the blank image generation and question answering. arXiv:1506.00278, 2015. [36] W. Zaremba and I. Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. [37] C. L. Zitnick, D. Parikh, and L. Vanderwende. Learning the visual interpretation of sentences. In ICCV, 2013. Ask Your Neurons: A Neural-based Approach to Answering Questions about Images - A Supplementary Material Mateusz Malinowski1 Marcus Rohrbach2 Mario Fritz1 1 Max Planck Institute for Informatics, Saarbrücken, Germany 2 UC Berkeley EECS and ICSI, Berkeley, CA, United States Malinowski et al. [1] Neural-Image-QA (ours) - multiple words - single word Human answers [1] Language only (ours) - multiple words - single word Human answers, no images Accuracy WUPS @0.9 WUPS @0.0 7.86 11.86 38.79 17.49 19.43 50.20 23.28 25.28 50.82 57.76 62.00 67.27 17.06 17.15 7.34 22.30 22.80 13.17 56.53 58.42 35.56 Table 1. Results on DAQUAR, all classes, single reference, in %. Replication of Table 1 from the main paper for convenience. In this supplemental material, we additionally provide qualitative examples of different variants of our architecture and show the correlations of predicted answer words and question words with human answer and question words. The examples are chosen to highlight challenges as well as differences between “Neural-Image-QA” and “Language only” architectures. Table 9 also shows a few failure cases. In all cases but “multiple words answer”, we use the best “single word” variants. Although “Language only” ignores the image, it is still able to make “reasonable guesses” by exploiting biases captured by the dataset that can be viewed as a type of common sense knowledge. For instance, “tea kettle” often sits on the oven, cabinets are usually “brown”, “chair” is typically placed in front of a table, and we commonly keep a “photo” on a cabinet (Table 2, 4, 5, 8). This effect is analysed in Figure 1. Each data point in the plot represents the correlation between a question and a predicted answer words for our “Language only” model (xaxis) versus the correlation in the human answers (y-axis). Despite the reasonable guesses of the “Language only” architecture, the “Neural-Image-QA” predicts in average better answers (shown in Table 1 that we have replicated Figure 1. Figure showing correlation between question and answer words of the “Language only” model (at x-axis), and a similar correlation of the “Human-baseline” [1] (at y-axis). from the main paper for the convenience of the reader) by exploiting the visual content of images. For instance in Table 6 the “Language only” model incorrectly answers “6” on the question “How many burner knobs are there ?” because it has seen only this answer during the training with exactly the same question but on different image. Both models, “Language only” and “Neural-ImageQA”, have difficulties to answer correctly on long questions or such questions that expect a larger number of answer words (Table 9). On the other hand both models are doing well on predicting a type of the question (e.g. “what color ...” result in a color name in the answer, or “how many ...” questions result in a number), there are a few rare cases with an incorrect type of the predicted answer (the last example in Table 9). References [1] M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. What are the objects close to the wall? Neural-Image-QA: What is on the stove? What is left of sink? wall decoration tea kettle tissue roll books tea kettle towel tea kettle tissue roll Language only: Ground truth answers: wall decoration Table 2. Examples of compound answer words. How many lamps are there? How many pillows are there on the bed? How many pillows are there on the sofa? Neural-Image-QA: 2 2 3 Language only: 2 3 3 Ground truth answers: 2 2 3 Table 3. Counting questions. What color is the towel? What color are the cabinets? What is the colour of the pillows? Neural-Image-QA: brown brown black, white Language only: white brown blue, green, red Ground truth answers: white brown black, red, white Table 4. Questions about color. What is hanged on the chair? What is the object close to the sink? What is the object on the table in the corner? Neural-Image-QA: clothes faucet lamp Language only: jacket faucet plant Ground truth answers: clothes faucet lamp Table 5. Correct answers by our “Neural-Image-QA” architecture. What are the things on the cabinet? What is in front of the shelf? How many burner knobs are there? Neural-Image-QA: photo chair 4 Language only: photo basket 6 Ground truth answers: photo chair 4 Table 6. Correct answers by our “Neural-Image-QA” architecture. What is the object close to the counter? What is the colour of the table and chair? How many towels are hanged? Neural-Image-QA: sink brown 3 Language only: stove brown 4 Ground truth answers: sink brown 3 Table 7. Correct answers by our “Neural-Image-QA” architecture. What is on the right most side on the table? Neural-Image-QA: Language only: Ground truth answers: What are the things on the coffee table? What is in front of the table? lamp books chair machine jacket chair lamp books chair Table 8. Correct answers by our “Neural-Image-QA” architecture. What is on the left side of the white oven on the floor and on right side of the blue armchair? Neural-Image-QA: Language only: Ground truth answers: What are the things on the cabinet? What color is the frame of the mirror close to the wardrobe? oven chair, lamp, photo pink exercise equipment candelabra curtain garbage bin lamp, photo, telephone white Table 9. Failure cases. arXiv:1505.01596v2 [cs.CV] 14 Sep 2015 Learning to See by Moving Pulkit Agrawal UC Berkeley João Carreira UC Berkeley Jitendra Malik UC Berkeley pulkitag@eecs.berkeley.edu carreira@eecs.berkeley.edu malik@eecs.berkeley.edu Abstract can be used to learn useful visual representations. However, until now unsupervised learning approaches [4, 22, 29, 32] have not yet delivered on their promise and are nowhere to be seen in current applications on complex real world imagery. The current dominant paradigm for feature learning in computer vision relies on training neural networks for the task of object recognition using millions of hand labelled images. Is it also possible to learn useful features for a diverse set of visual tasks using any other form of supervision ? In biology, living organisms developed the ability of visual perception for the purpose of moving and acting in the world. Drawing inspiration from this observation, in this work we investigate if the awareness of egomotion can be used as a supervisory signal for feature learning. As opposed to the knowledge of class labels, information about egomotion is freely available to mobile agents. We show that using the same number of training images, features learnt using egomotion as supervision compare favourably to features learnt using class-label as supervision on the tasks of scene recognition, object recognition, visual odometry and keypoint matching. Biological agents use perceptual systems for obtaining sensory information about their environment that enables them to act and accomplish their goals [13, 9]. Both biological and robotic agents employ their motor system for executing actions in their environment. Is it also possible that these agents can use their own motor system as a source of supervision for learning useful perceptual representations? Motor theories of perception have a long history [13, 9], but there has been little work in formulating computational models of perception that make use of motor information. In this work we focus on visual perception and present a model based on egomotion (i.e. self motion) for learning useful visual representations. When we say useful visual representations [34], we mean representations that possess the following two characteristics - (1) ability to perform multiple visual tasks and (2) ability of performing new visual tasks by learning from only a few labeled examples provided by an extrinsic teacher. ”We move in order to see and we see in order to move” J.J Gibson Mobile agents are naturally aware of their egomotion (i.e. self-motion) through their own motor system. In other words, knowledge of egomotion is “freely” available. For example, the vestibular system provides the sense of orientation in many mammals. In humans and other animals, the brain has access to information about eye movements and the actions performed by the animal [9]. A mobile robotic agent can estimate its egomotion either from the motor commands it issues to move or from odometry sensors such as gyroscopes and accelerometers mounted on the agent itself. 1. Introduction Recent advances in computer vision have shown that visual features learnt by neural networks trained for the task of object recognition using more than a million labelled images are useful for many computer vision tasks like semantic segmentation, object detection and action classification [18, 10, 1, 33]. However, object recognition is one among many tasks for which vision is used. For example, humans use visual perception for recognizing objects, understanding spatial layouts of scenes and performing actions such as moving around in the world. Is there something special about the task of object recognition or is it the case that useful visual representations can be learnt through other modes of supervision? Clearly, biological agents perform complex visual tasks and it is unlikely that they require external supervision in form of millions of labelled examples. Unlabelled visual data is freely available and in theory this data We propose that useful visual representations can be learnt by performing the simple task of correlating visual stimuli with egomotion. A mobile agent can be treated like a camera moving in the world and thus the knowledge of egomotion is the same as the knowledge of camera motion. Using this insight, we pose the problem of correlating visual stimuli with egomotion as the problem of predicting the camera transformation from the consequent pairs of im1 ages that the agent receives while it moves. Intuitively, the task of predicting camera transformation between two images should force the agent to learn features that are adept at identifying visual elements that are present in both the images (i.e. visual correspondence). In the past, features such as SIFT, that were hand engineered for finding correspondences were also found to be very useful for tasks such as object recognition [23, 19]. This suggests that egomotion based learning can also result in features that are useful for such tasks. In order to test our hypothesis of feature learning using egomotion, we trained multilayer neural networks to predict the camera transformation between pairs of images. As a proof of concept, we first demonstrate the usefulness of our approach on the MNIST dataset [21]. We show that features learnt using our method outperform previous approaches of unsupervised feature learning when class-label supervision is available only for a limited number of examples (section 3.4) Next, we evaluated the efficacy of our approach on real world imagery. For this purpose, we used image and odometry data recorded from a car moving through urban scenes, made available as part of the KITTI [12] and the San Francisco (SF) city [7] datasets. This data mimics the scenario of a robotic agent moving around in the world. The quality of features learnt from this data were evaluated on four tasks (1) Scene recognition on SUN [38] (section 5.1), (2) Visual odometery (section 5.4), (3) Keypoint matching (section 5.3) and (4) Object recognition on Imagenet [31] (section 5.2). Our results show that for the same amount of training data, features learnt using egomotion as supervision compare favorably to features learnt using class-label as supervision. We also show that egomotion based pretraining outperforms a previous approach based on slow feature analysis for unsupervised learning from videos [37, 14, 25]. To the best of our knowledge, this work provides the first effective demonstration of learning visual representations from non-visual access to egomotion information in real world setting. The rest of this paper is organized as following: In section 2 we discuss the related work, in section 3, 4, 5 we present the method, dataset details and we conclude with the discussion in section 6. of learnt features. Despite a lot of work in unsupervised learning (see [4] for a review), a method that works on complex real world imagery is yet to be developed. An alternative to unsupervised learning is to learn features using intrinsic reward signals that are freely available to a system (i.e self-supervised learning). For instance, [15] used intrinsic reward signals available to a robot for learning features that predict path traversability, while [28] trained neural networks for driving vehicles directly from visual input. In this work we propose to use non-visual access to egomotion information as a form of self-supervision for visual feature learning. Unlike any other previous work, we show that our method works on real world imagery. Closest to our method is the the work of transforming auto-encoders [16] that used egomotion to reconstruct the transformed image from an input source image. This work was purely conceptual in nature and the quality of learned features was not evaluated. In contrast, our method uses egomotion as supervision by predicting the transformation between two images using a siamese-like network model [8]. Our method can also be seen as an instance of feature learning from videos. [37, 14, 25] perform feature learning from videos by imposing the constraint that temporally close frames should have similar feature representations (i.e. slow feature analysis) without accounting for either the camera motion or the motion of objects in the scene. In many settings the camera motion dominates the motion content of the video. Our key observation is that knowledge of camera motion (i.e. egomotion) is freely available to mobile agents and can be used as a powerful source of self-supervision. 3. A Simple Model of Motion-based Learning We model the visual system of the agent with a Convolutional Neural Network (CNN, [20]). The agent optimizes its visual representations (i.e. updating the weights of the CNN) by minimizing the error between the egomotion information (i.e. camera transformation) obtained from its motor system and egomotion predicted using its visual inputs only. Performing this task is equivalent to training a CNN with two streams (i.e. Siamese Style CNN or SCNN[8]) that takes two images as inputs and predicts the egomotion that the agent underwent as it moved between the two spatial locations from which the two images were obtained. In order to learn useful visual representations, the agent continuously performs this task as it moves around in its environment. In this work we use the pretraining-finetuning paradigm for evaluating the utility of learnt features. Pretraining is the process of optimizing the weights of a randomly initialized CNN for an auxiliary task that is not the same as the target task. Finetuning is the process of modifying the weights of 2. Related Work Past work in unsupervised learning has been dominated by approaches that pose feature learning as the problem of discovering compact and rich representations of images that are also sufficient to reconstruct the images [6, 3, 22, 32, 27, 30]. Another line of work has focused on learning features that are invariant to transformations either from video [37, 14, 25] or from images [11, 29]. [24] perform feature learning by modeling spatial transformations using boltzmann machines, but donot evaluate the quality 2 Figure 1: Exploring the utility of egomotion as supervision for learning useful visual features. A mobile agent equipped with visual sensors receives a sequence of images as inputs while it moves in its environment. The movement of the agent is equivalent to the movement of a camera. In this work, egomotion based learning is posed as the problem of predicting camera transformation from image pairs. The top and bottom rows of the figure show some sample image pairs from the SF and KITTI datasets that were used for feature learning. Base-CNN Stream-1 L1 L2 Lk F1 3.1. Two Stream Architecture Each stream of the CNN independently computes features for one image. Both streams share the same architecture and the same set of weights and consequently perform the same set of operations for computing features. The individual streams have been called as BaseCNN (BCNN). Features from two BCNNs are concatenated and passed downstream into another CNN called as the TopCNN (TCNN) (see figure 2). TCNN is responsible for using the BCNN features to predict the camera transformation between the input pair of images. After pretraining, the TCNN is removed and a single BCNN is used as a standard CNN for feature computation for the target task. Base-CNN Stream-2 F2 Transformation a pretrained CNN for the given target task. Our experiments compare the utility of features learnt using egomotion based pretraining against class-label based and slow-feature based pretraining on multiple target tasks. Top-CNN Figure 2: Description of the method for feature learning. Visual features are learnt by training a Siamese style Convolutional Neural Network (SCNN, [8]) that takes as inputs two images and predicts the transformation between the images (i.e. egomotion). Each stream of the SCNN (called as Base-CNN or BCNN) computes features for one image. The outputs of two BCNNs are concatenated and passed as inputs to a second multilayer CNN called as the TopCNN (TCNN) (shown as layers F1 , F2 ). The two BCNNs have the same architecture and share weights. After feature learning, TCNN is discarded and a single BCNN stream is used as a standard CNN for extracting features for performing target tasks like scene recognition. 3.2. Shorthand for CNN architectures The abbreviations Ck, Fk, P, D, Op represent a convolutional(C) layer with k filters, a fully-connected(F) layer with k filters, pooling(P), dropout(D) and the output(Op) layers respectively. We used ReLU non-linearity after every convolutional/fully-connected layer, except for the output layer. The dropout layer was always used with dropout of 0.5. The output layer was a fully connected layer with number of units equal to the number of desired outputs. As an example of our notation, C96-P-F500-D refers to a network with 96 filters in the convolution layer followed by ReLU non-linearity, a pooling layer, a fully-connected layer with 500 unit, ReLU non-linearity and a dropout layer. We used [17] for training all our models. 3.3. Slow Feature Analysis (SFA) Baseline Slow Feature Analysis (SFA) is a method for feature learning based on the principle that useful features change 3 Table 1: Comparison of various pretraining methods on MNIST reveals that egomotion based pretraining outperforms many previous approaches for unsupervised learning. The performance is reported as the error rate. slowly in time. We used the following contrastive loss formulation of SFA [8, 25], L(xt1 , xt2 , W ) = ( D(xt1 , xt2 )
1 − max 0, m − D(xt1 , xt2 ) if |t1 − t2 | ≤ T if |t1 − t2 | > T Method (1) Autoencoder [17] Ranzato et al. [29] Lee et al. [22] Train from Scratch SFA (m=10) SFA (m=100) Egomotion (ours) where, L is the loss, xt1 , xt2 refer to feature representations of frames observed at times t1 , t2 respectively, W are the parameters that specify the feature extraction process, D is a measure of distance with parameter, m is a predefined margin and T is a predefined time threshold for determining whether the two frames are temporally close or not. In this work, xt are features computed using a CNN with weights W and D was chosen to be the L2 distance. SFA pretraining was performed using two stream architectures that took pairs of images as inputs and produced outputs xt1 , xt2 as outputs from the two streams respectively. # examples for finetuning 100 300 1000 10000 24.1 12.2 7.7 4.8 - 7.18 3.21 0.85 - 2.62 20.1 8.3 4.5 1.6 11.2 6.4 3.5 2.1 11.9 6.4 4.8 4.7 8.7 3.6 2.0 0.9 3.4. Proof of Concept using MNIST Pretraining was performed for 40K iterations (i.e. 5M examples) using an initial learning rate of 0.01 which was reduced by a factor of 2 after every 10K iterations. On MNIST, egomotion was emulated by generating synthetic data consisting of random transformation (translations and rotations) of digit images. From the training set of 60K images, digits were randomly sampled and then transformed using two different sets of random transformations to generate image pairs. CNNs were trained for predicting the transformations between these image pairs. The following architecture was used for finetuning: BCNN-F500-D-Op. In order to evaluate the quality of BCNN features, the learning rate of all layers in the BCNN were set to 0 during finetuning for digit classification. Finetuning was performed for 4K iterations (which is equivalent to training for 50 epochs for the 10K labelled training examples) with a constant learning rate of 0.01. 3.4.1 Data For egomotion based pretraining, relative translation between the digits was constrained to be an integer value in the range [-3, 3] and relative rotation θ was constrained to lie within the range [-30◦ , 30◦ ]. The prediction of transformation was posed as a classification task with three separate soft-max losses (one each for translation along X, Y axes and the rotation about Z-axis). SCNN was trained to minimize the sum of these three losses. Translations along X, Y were separately binned into seven uniformly spaced bins each. The rotations were binned into bins of size 3◦ each resulting into a total of 20 bins (or classes). For SFA based pretraining, image pairs with relative translation in the range [-1, 1] and relative rotation within [-3◦ , 3◦ ] were considered to be temporally close to each other (see equation 1). A total of 5 million image pairs were used for both pretraining procedures. 3.4.2 3.4.3 Results The BCNN features were evaluated by computing the error rates on the task of digit classification using 100, 300, 1K and 10K class-labelled examples for training. These sets were constructed by randomly sampling digits from the standard training set of 60K digits. For this part of the experiment, the original digit images were used (i.e. without any transformations or data augmentation). The standard test set of 10K digits was used for evaluation and error rates averaged across 3 runs are reported in table 1. The BCNN architecture: C96-P-C256-P, was found to be optimal for egomotion and SFA based pretraining and also for training from scratch (i.e. random weight initialization). Results for other architectures are provided in the supplementary material. For SFA based pretraining, we experimented with multiple values of the margin m and found that m = 10, 100 led to the best performance. Our method outperforms convolutional deep belief networks [22], a previous approach based on learning features invariant to transformations [29] and SFA based pretraining. Network Architectures We experimented with multiple BCNN architectures and chose the optimal architecture for each pretraining method separately. For egmotion based pretraining, the two BCNN streams were concatenated using the TCNN: F1000-D-Op. 4 4. Learning Visual Features From Egomotion in Natural Environments We used two main sources of real world data for feature learning: the KITTI and SF datasets, which were collected using cameras and odometry sensors mounted on a car driving through urban scenes. Details about the data, the experimental procedure, the network architectures and the results are provided in sections 4.1, 4.2, 4.3 and 5 respectively. (a) KITTI-Net (b) SF-Net Figure 3: Visualization of layer 1 filters learnt by egomotion based pretraining on (a) KITTI and (b) SF datasets. A large majority of layer-1 filters are color detectors and some of them are edge detectors. This is expected as color is a useful cue for determining correspondences between image pairs. 4.1. KITTI Dataset The KITTI dataset provided odometry and image data recorded during 11 short trips of variable length made by a car moving through urban landscapes. The total number of frames in the entire dataset was 23,201. Out of 11, 9 sequences were used for training and 2 for validation. The total number of images in the training set was 20,501. The odometry data was used to compute the camera transformation between pairs of images recorded from the car. The direction in which the camera pointed was assumed to be the Z axis and the image plane was taken to be the XY plane. X-axis and Y-axis refer to horizontal and vertical directions in the image plane. As significant camera transformations in the KITTI data were either due to translations along the Z/X axis or rotation about the Y axis, only these three dimensions were used to express the camera transformation. The rotation was represented as the euler angle about the Y-axis. The task of predicting the transformation between pair of images was posed as a classification problem. The three dimensions of camera transformation were individually binned into 20 uniformly spaced bins each. The training image pairs were selected from frames that were at most ±7 frames apart to ensure that images in any given pair would have a reasonable overlap. For SFA based pretraining, pairs of frames that were separated by atmost ±7 frames were considered to be temporally close to each other. The SCNN was trained to predict camera transformation from pairs of 227 × 227 pixel sized image regions extracted from images of overall size 370 × 1226 pixels. For each image pair, the coordinates for cropping image regions were randomly chosen. Figure 1 illustrates typical image crops. and the 3 translations). Since, it is unreasonable to expect that visual features can be used to infer big camera transformations, rotations between [-30◦ , 30◦ ] were binned into 10 uniformly spaced bins and two extra bins were used for rotations larger and smaller than 30◦ and -30◦ respectively. The three translations were individually binned into 10 uniformly spaced bins each. Images were resized to a size of 360 × 480 and image regions of size 227 × 227 were used for training the SCNN. 4.3. Network Architecture BCNN closely followed the architecture of first five AlexNet layers [18]: C96-P-C256-P-C384-C384-C256-P. TCNN architecture was: C256-C128-F500-D-Op. The convolutional filters in the TCNN were of spatial size 3×3. The networks were trained for 60K iterations with a batch size of 128. The initial learning rate was set to 0.001 and was reduced by a factor of two after every 20K iterations. We term the networks pretrained using egomotion on KITTI and SF datasets as KITTI-Net and SF-Net respectively. The net pretrained on KITTI with SFA is called KITTI-SFA-Net. Figure 3 shows the layer-1 filters of KITTI-Net and SF-Net. A large majority of layer-1 filters are color detectors, while some of them are edge detectors. As color is a useful cue for determining correspondences between closeby frames of a video sequence, learning of color detectors as layer-1 filters is not surprising. The fraction of filters that detect edges is higher for the SF-Net. This is not surprising either, because higher fraction of images in the SF dataset contain structured objects like buildings and cars. 4.2. SF Dataset SF dataset provides camera transformation between ≈ 136K pairs of images (constructed from a set of 17,357 unique images). This dataset was constructed using Google StreetView [7]. ≈ 130K image pairs were used for training and ≈ 6K pairs for validation. Just like KITTI, the task of predicting camera transformation was posed as a classification problem. Unlike KITTI, significant camera transformation was found along all six dimensions of transformation (i.e. the 3 euler angles 5. Evaluating Motion-based Learning For evaluating the merits of the proposed approach, features learned using egomotion based supervision were compared against features learned using class-label and SFA based supervision on the challenging tasks of scene recogni5 tion, intra-class keypoint matching and visual odometry and object recognition. The ultimate goal of feature learning is to find features that can generalize from only a few supervised examples on a new task. Therefore it makes sense to evaluate the quality of features when only a few labelled examples for the target task are provided. Consequently, the scene and object recognition experiments were performed in the setting when only 1-20 labelled examples per class were available for finetuning. The KITTI-Net and SF-Net (examples of models trained using egomotion based supervision) were trained using only only ≈ 20K unique images. To make a fair comparison with class-label based supervision, a model with AlexNet architecture was trained using only 20K images taken from the training set of ILSVRC12 challenge (i.e. 20 examples per class). This model has been referred to as AlexNet-20K. In addition, some experiments presented in this work also make comparison with AlexNet models trained with 100K and 1M images that have been named as AlexNet-100K and AlexNet-1M respectively. of roads, buildings, cars, few pedestrians, trees and some vegetation. It is in fact surprising that a network trained on data with such little diversity is competitive on classifying indoor and outdoor scenes with the AlexNet-20K that was trained on a much more diverse set of images. We believe that with more diverse training data for egomotion based learning, the performance of learnt features will be better than currently reported numbers. The KITTI-Net outperformed the SF-Net except for the performance of layer 1 (L1). As it was possible to extract a larger number of image region pairs from the KITTI dataset as compared to the SF dataset (see section 4.1, 4.2), the result that KITTI-Net outperforms SF-Net is not surprising. Because KITTI-Net was found to be superior to the SF-Net in this experiment, the KITTI-Net was used for all other experiments described in this paper. 5.2. Object Recognition If egomotion based pretraining learns useful features for object recognition, then a net initialized with KITTI-Net weights should outperform a net initialized with random weights on the task of object recognition. For testing this, we trained CNNs using 1, 5, 10 and 20 images per class from the ILSVRC-2012 challenge. As this dataset contains 1000 classes, the total number of training examples available for training for these networks were 1K, 5K, 10K and 20K respectively. All layers of KITTI-Net, KITTI-SFA-Net and AlexNet-Scratch (i.e. CNN with random weight initialization) were finetuned for image classification. The results of the experiment presented in table 3 show that egomotion based supervision (KITTI-Net) clearly outperforms SFA based supervision(KITTI-SFA-Net) and AlexNet-Scratch. As expected, the improvement offered by motion-based pretraining is larger when the number of examples provided for the target task are fewer. These result show that egomotion based pretraining learns features useful for object recognition. 5.1. Scene Recognition SUN dataset consisting of 397 indoor/outdoor scene categories was used for evaluating scene recognition performance. This dataset provides 10 standard splits of 5 and 20 training images per class and a standard test set of 50 images per class. Due to time limitation of running 10 runs of the experiment, we evaluated the performance using only 3 train/test splits. For evaluating the utility of CNN features produced by different layers, separate linear (SoftMax) classifiers were trained on features produced by individual CNN layers (i.e. BCNN layers of KITTI-Net, KITTI-SFA-Net and SFNet). Table 2 reports recognition accuracy (averaged over 3 train/test splits) for various networks considered in this study. KITTI-Net outperforms SF-Net and is comparable to AlexNet-20K. This indicates that given a fixed budget of pretraining images, egomotion based supervision learns features that are almost as good as the features using classbased supervision on the task of scene recognition. The performance of features computed by layers 1-3 (abbreviated as L1, L2, L3 in table 2) of the KITTI-SFA-Net and KITTINet is comparable, whereas layer 4, 5 features of KITTI-Net significantly outperform layer 4, 5 features of KITTI-SFANet. This indicates that egomotion based pretraining results into learning of higher-level features, while SFA based pretraining results into learning of lower-level features only. The KITTI-Net outperforms GIST[26], which was specifically developed for scene classification, but is outperformed by Dense SIFT with spatial pyramid matching (SPM) kernel [19]. The KITTI-Net was trained using limited visual data (≈ 20Kf rames) containing visual imagery of limited diversity. The KITTI data mainly contains images 5.3. Intra-Class Keypoint Matching Identifying the same keypoint of an object across different instances of the same object class is an important visual task. Visual features learned using egomotion, SFA and class-label based supervision were evaluated for this task using keypoint annotations on the PASCAL dataset [5]. Keypoint matching was computed in the following way: First, ground-truth object bounding boxes (GT-BBOX) from PASCAL-VOC2012 dataset were extracted and resized (while preserving the aspect ratio) to ensure that the smaller side of the boxes was of length 227 pixels. Next, feature maps from layers 2-5 of various CNNs were computed for every GT-BBOX. The keypoint matching score was computed between all pairs of GT-BBOX belonging to the same object class. For given pair of GT-BBOX, the fea6 Table 2: Comparing the accuracy of neural networks pre-trained using motion-based and class-label based supervision for the task of scene recognition on the SUN dataset. The performance of layers 1-6 (labelled as L1-L6) of these networks was evaluated after finetuning the network using 5/20 images per class from the SUN dataset. The performance of the KITTI-Net (i.e. motion-based pretraining) fares favorably with a network pretrained on Imagenet (i.e. class-based pretraining) with the same number of pretraining images (i.e. 20K). Method Pretrain Supervision #Pretrain #Finetune L1 L2 L3 L4 L5 L6 #Finetune L1 L2 L3 L4 L5 L6 AlexNet-1M 1M 5 5.3 10.5 12.1 12.5 18.0 23.6 20 11.8 22.2 25.0 26.8 33.3 37.6 Class-Label AlexNet-20K 20K 5 4.9 6.3 6.6 6.3 6.6 6.7 20 8.7 12.6 12.4 11.9 12.5 12.4 KITTI-SFA-Net Slowness 20.5K 5 4.5 5.7 6.2 3.4 0.5 20 8.2 11.2 12.0 7.3 1.1 SF-Net 18K 5 4.4 5.2 4.9 5.1 4.7 20 8.6 11.6 10.9 10.4 9.1 Egomotion KITTI-Net 20.5K 5 4.3 6.0 5.9 5.8 6.4 20 7.9 12.2 12.1 11.7 12.4 GIST [38] Human 5 6.2 20 11.6 SPM [38] Human 5 8.4 20 16.0 Table 3: Top-5 accuracy on the task of object recognition on the ILSVRC-12 validation set. AlexNet-Scratch refers to a net with AlexNet architecture initialized with randomly weights. The weights of KITTI-Net and KITTI-SFA-Net were learned using egomotion based and SFA based supervision on the KITTI dataset respectively. All the networks were finetuned using 1, 5, 10, 20 examples per class. The KITTI-Net clearly outperforms AlexNet-Scratch and KITTI-SFA-Net. Method AlexNet-Scratch KITTI-SFA-Net (Slowness) KITTI-Net (Egomotion) 1 1.1 1.5 2.3 5 3.1 3.9 5.1 10 5.9 6.1 8.6 Table 4: Comparing the accuracy of various pretraining methods on the task of visual odometry. Method Translation Acc. δX δY δZ SF-Net 40.2 58.2 38.4 KITTI-Net 43.4 57.9 40.2 AlexNet-1M 41.8 58.0 39.0 20 14.1 14.9 15.8 Rotation Acc. δθ1 δθ2 δθ3 45.0 44.8 40.5 48.4 44.0 41.0 46.0 44.5 40.5 nation for this observation is that with only 20K examples, features learnt by AlexNet-20K only capture coarse global appearance of objects and are therefore poor at keypoint matching. SIFT has been hand engineered for finding correspondences across images and performs as well as the best AlexNet-1M features for this task (i.e. conv-4 features). KITTI-Net also significantly outperforms KITTISFA-Net. These results indicate that features learnt by egomotion based pretraining are superior to SFA and classlabel based pretraining for the task of keypoint matching. tures associated with keypoints in the first image were used to predict the location of the same keypoints in the second image. The normalized pixel distance between the actual and predicted keypoint locations was taken as the error in keypoint matching. More details about this procedure have been provided in the supp. materials. It is natural to expect that accuracy of keypoint matching would depend on the camera transformation between the two viewpoints of the object(i.e. viewpoint distance). In order to make a holistic evaluation of the utility of features learnt by different pretraining methods on this task, matching error was computed as a function of viewpoint distance [36]. Figure 4 reports the matching error averaged across all keypoints, all pairs of GT-BBOX and all classes using features extracted from layers conv-3 and conv-4. KITTI-Net trained only with 20K unique frames was superior to AlexNet-20K and AlexNet-100K and inferior only to AlexNet-1M. A net with AlexNet architecture initialized with random weights (AlexNet-Rand), surprisingly performed better than AlexNet-20K. One possible expla- 5.4. Visual Odometry Visual odometry is the task of estimating the camera transformation between image pairs. All layers of KITTINet and AlexNet-1M were finetuned for 25K iterations using the training set of SF dataset on the task of visual odometry (see section 4.2 for task description). The performance of various CNNs was evaluated on the validation set of SF dataset and the results are reported in table 4. Performance of KITTI-Net was either superior or comparable to AlexNet-1M on this task. As the evaluation was made on the SF dataset itself, it was not surprising that on some metrics SF-Net outperformed KITTI-Net. The results of this experiment indicate that egomotion based feature learning is superior to class-label based feature learning on the task of visual odometry. 7 0.27 0.27 0.25 0.25 0.23 Mean error Mean error 0.29 0.23 KittiNet-20k-cv3 KittiNet-SFA-20k-cv3 AlexNet-rand-cv3 AlexNet-20k-cv3 AlexNet-100k-cv3 AlexNet-1M-cv3 SIFT 0.21 0.19 0.17 0 18 36 54 72 90 108 126 144 0.21 KittiNet-20k-cv4 KittiNet-SFA-20k-cv4 AlexNet-rand-cv4 AlexNet-20k-cv4 AlexNet-100k-cv4 AlexNet-1M-cv4 SIFT 0.19 0.17 0.15 162 0 18 Maximum viewpoint range 36 54 72 90 108 126 144 162 Maximum viewpoint range Figure 4: Intra-class keypoint matching error as a function of viewpoint distance averaged over 20 PASCAL objects using features from layers conv3 (left) and conv4 (right) of various CNNs used in this work. Please see the text for more details. 6. Discussion unable to evaluate the utility of motion-based supervision across the full spectrum of training set sizes. In this work, we chose to first pretrain our models using a base task (i.e. egomotion) and then finetune these models for target tasks. An equally interesting setting is that of online learning where the agent has continuous access to intrinsic supervision (such as egomotion) and occasional explicit access to extrinsic teacher signal (such as the class labels). We believe that such a training procedure is likely to result in learning of better features. Our intuition behind this is that seeing different views of the same instance of an object (say) car, may not be sufficient to learn that different instances of the car class should be grouped together. The occasional extrinsic signal about object labels may prove useful for the agent to learn such concepts. Also, current work makes use of passively collected egomotion data and it would be interesting to investigate if it is possible to learn better visual representations if the agent can actively decide on how to explores its environment (i.e. active learning [2]). In this work, we have shown that egomotion is a useful source of intrinsic supervision for visual feature learning in mobile agents. In contrast to class labels, knowledge of egomotion is ”freely” available. On MNIST, egomotion-based feature learning outperforms many previous unsupervised methods of feature learning. Given the same budget of pretraining images, on task of scene recognition, egomotionbased learning performs almost as well as class-label-based learning. Further, egomotion based features outperform features learnt by a CNN trained using class-label supervision on two orders of magnitude more data (AlexNet-1M) on the task of visual odometry and one order of magnitude more data on the task of intra-class keypoint matching. In addition to demonstrating the utility of egomotion based supervision, these results also suggest that features learnt by class-label based supervision are not optimal for all visual tasks. This means that future work should look at what kinds of pretraining are useful for what tasks. One potential criticism of our work is that we have trained and evaluated high capacity deep models on relatively little data (e.g. only 20K unique images available on the KITTI dataset). In theory, we could have learnt better features by downsizing the networks. For example, in our experiments with MNIST we found that pretraining a 2-layer network instead of 3-layer results in better performance (table 1). In this work, we have made a conscious choice of using standard deep models because the main goal of this work was not to explore novel feature extraction architectures but to investigate the value of egmotion for learning visual representations on architectures known to perform well on practical applications. Future research focused on exploring architectures that are better suited for egomotion based learning can only make a stronger case for this line of work. While egomotion is freely available to mobile agents, there are currently no publicly available datasets as large as Imagenet. Consequently, we were Acknowledgements This work was supported in part by ONR MURIN00014-14-1-0671. Pulkit Agrawal was partially supported by Fulbright Science and Technology Fellowship. João Carreira was supported by the Portuguese Science Foundation, FCT, under grant SFRH/BPD/84194/2012. We gratefully acknowledge NVIDIA corporation for the donation of Tesla GPUs for this research. Appendix A. Keypoint Matching Score Consider images of two instances of the same object class (for example airplane images as shown in first row of figure 5) for which keypoint matching score needs to be computed. The images are pre-processed in the following way: 8 • Crop the groundtruth bounding box from the image. A.2. Keypoint Matching using SIFT • Pad the images by 30 pixels along each dimension. SIFT features are extracted using a square window of size 72 pixels and a stride of 8 pixels using the open source code from [35]. The stride of 8 pixels was chosen to have a fair comparison with the CNN features. The CNN features were computed with a stride of 8 for layer conv-2 and stride of 16 for layers conv-3, conv-4 and conv-5 respectively. The matching error using SIFT was calculated in the same way as for the CNNs. • Resize each image so that the smallest side is 227 pixels. The aspect ratio of the image is preserved. A.1. Keypoint Matching using CNN Assume that the lth layer of the CNN is used for feature computation. The feature map produced by the lth layer is of dimensionality I × J × M , where (I, J) are the spatial dimensions and M is the number of filters in the lth layer. Thus, the lth layer produces a M dimensional feature vector for each of the I × J grid position in the feature map. The coordinates of the keypoints are provided in the image coordinate system [5]. For the keypoints in the first image, we first determine their grid position in the I × J feature map. Each grid position has an associated receptive field in the image. The keypoints are assigned to the grid positions for which the center of receptive field is closest to the keypoints. This means that each keypoint is assigned one location in the feature map. Let the M dimensional feature vector associated with the k th keypoint in the first image be F1k . Let the M dimensional feature vector at grid location Cij for the second image be F2 (Cij ). The location of matching keypoint in the second image is determined by solving: C∗ = argminCij kF1k − F2 (Cij )k2 A.3. Effect of Viewpoint on Keypoint Matching Intuitively, matching instances of the same object that are related by a large transformation (i.e. viewpoint distance) should be harder than matching instances with a small viewpoint distance. Therefore, in order to obtain a holistic understanding of the accuracy of features in performing keypoint matching it is instructive to study the accuracy of matching as a function of viewpoint distance. [36] aligned instances of the same class (from PASCALVOC-2012) in a global coordinate system and provide a rotation matrix (R) for each instance in the class. To measure the viewpoint distance, we computed the riemannian metric on the manifold of rotation matrices ||log(Ri RjT )||F , where log is the matrix logarithm, ||.||F is the Frobenius norm of the matrix and Ri , Rj are the rotation matrices for the ith , j th instances respectively. We binned the distances into 10 uniform bins (of 18◦ each). In Figure 4 of the main paper we show the mean error in keypoint matching in each of these viewpoints bin. The matching error in the k th bin is calculated by considering all the instances with a viewpoint distance ≤ k×18◦ , for k ∈ [1, 10]. As expected we find that keypoint matching is worse for larger viewpoint distances. (2) C∗ is transformed into the image coordinate system by computing the center of receptive field (in the image) associated with this grid position. Let this transformed coordinates be C∗im and the coordinates of the corresponding keypoint (in im . The matching error for the k th the second image) be Cgt keypoint (Ek ) is defined as: im kC∗im − Cgt k2 Ek = L2D A.4. Matching Error for layers 2 and 5 (3) References L2D is the length of diagonal (in pixels) of the second where, image. As different images have different sizes, dividing by L2D normalizes for the difference in sizes. The matching error for a pair of images of instances belonging to the same class is calculated as: PK Ek Einstance = k=1 (4) K The average matching error across all pairs of the instance of the same class is given by Eclass : P Einstance Eclass = instance (5) #pairs [1] P. Agrawal, R. Girshick, and J. Malik. Analyzing the performance of multilayer neural networks for object recognition. In Computer Vision–ECCV 2014, pages 329–344. Springer, 2014. [2] R. Bajcsy. Active perception. Proceedings of the IEEE, 76(8):966–1005, 1988. [3] H. Barlow. Unsupervised learning. Neural computation, 1(3):295–311, 1989. [4] Y. Bengio, A. C. Courville, and P. Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.5538, 1, 2012. [5] L. Bourdev, S. Maji, T. Brox, and J. Malik. Detecting people using mutually consistent poselet activations. In Computer Vision–ECCV 2010, pages 168–181. Springer, 2010. [6] H. Bourlard and Y. Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4-5):291–294, 1988. where, #pairs is the number of pairs of object instances belonging to the same class. In Figure 4 of the main paper we report the matching error averaged across all the 20 classes. 9 AlexNet-20K AlexNet-100K AlexNet-1M KittiNet-20K SIFT Figure 5: Example matchings between pairs of objects (randomly chosen) with viewpoints within 60 degrees of each other, for classes ”aeroplane”, ”bottle”, ”dog”, ”person” and ”tvmonitor” from PASCAL VOC. The matchings have been shown for features from layer conv-4 of AlexNet-20K, AlexNet-100K, AlexNet-1M, KittiNet-20K and SIFT. The left image shows the ground truth keypoints that were matched with the keypoints in the right image. Right images shows the location of the ground truth keypoint (shown by solid dot) and lines joining the predicted keypoint location (tip of the line) with the ground keypoint location. Please see section A for details of keypoint matching procedure and figure 4 in the main paper for numerical results. This figure is best seen in color and with zoom. 10 0.29 0.29 0.27 0.27 Mean error 0.23 KittiNet-20k-p2 KittiNet-SFA-20k-p2 AlexNet-rand-p2 AlexNet-20k-p2 AlexNet-100k-p2 AlexNet-1M-p2 SIFT 0.21 0.19 0.17 0 18 36 54 72 90 108 126 144 Mean error 0.25 0.25 0.23 KittiNet-20k-cv5 KittiNet-SFA-20k-cv5 AlexNet-rand-cv5 AlexNet-20k-cv5 AlexNet-100k-cv5 AlexNet-1M-cv5 SIFT 0.21 0.19 0.17 162 0 Maximum viewpoint range 18 36 54 72 90 108 126 144 162 Maximum viewpoint range Figure 6: Intra-class keypoint matching error as a function of viewpoint distance averaged over 20 PASCAL objects using features extracted from layers pool-2 (left) and conv-5 (right) of various networks used in this work. Please see section 5.3 for more details. [7] D. M. Chen, G. Baatz, K. Koser, S. S. Tsai, R. Vedantham, T. Pylvanainen, K. Roimela, X. Chen, J. Bach, M. Pollefeys, et al. City-scale landmark identification on mobile devices. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 737–744, 2011. [8] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 539–546. IEEE, 2005. [9] J. E. Cutting. Perception with an eye for motion, volume 177. [10] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531, 2013. [11] P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor matching with convolutional neural networks: a comparison to sift. arXiv preprint arXiv:1405.5769, 2014. [12] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013. [13] J. J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, 1979. [14] R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun. Unsupervised feature learning from temporal data. arXiv preprint arXiv:1504.02518, 2015. [15] R. Hadsell, P. Sermanet, J. Ben, A. Erkan, J. Han, B. Flepp, U. Muller, and Y. LeCun. Online learning for offroad robots: Using spatial label propagation to learn long-range traversability. In Proc. of Robotics: Science and Systems (RSS), volume 11, page 32, 2007. [16] G. E. Hinton, A. Krizhevsky, and S. D. Wang. Transforming auto-encoders. In Artificial Neural Networks and Machine Learning–ICANN, pages 44–51. Springer, 2011. [17] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of [18] [19] [20] [21] [22] [23] [24] [25] [26] 11 the ACM International Conference on Multimedia, pages 675–678. ACM, 2014. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012. S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 2, pages 2169–2178. IEEE, 2006. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 609–616. ACM, 2009. D. G. Lowe. Object recognition from local scale-invariant features. In Computer vision, 1999. The proceedings of the seventh IEEE international conference on, volume 2, pages 1150–1157. Ieee, 1999. R. Memisevic and G. E. Hinton. Learning to represent spatial transformations with factored higher-order boltzmann machines. Neural Computation, 22(6):1473–1492, 2010. H. Mobahi, R. Collobert, and J. Weston. Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 737–744. ACM, 2009. A. Oliva and A. Torralba. Building the gist of a scene: The role of global image features in recognition. Progress in brain research, 155:23–36, 2006. [27] B. A. Olshausen et al. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996. [28] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Technical report, DTIC Document, 1989. [29] M. Ranzato, F. J. Huang, Y.-L. Boureau, and Y. LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pages 1–8. IEEE, 2007. [30] M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604, 2014. [31] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015. [32] R. Salakhutdinov and G. E. Hinton. Deep boltzmann machines. In International Conference on Artificial Intelligence and Statistics, pages 448–455, 2009. [33] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, pages 568–576, 2014. [34] S. Soatto. Visual scene representations: Sufficiency, minimality, invariance and approximations. arXiv preprint arXiv:1411.7676, 2014. [35] A. Vedaldi and B. Fulkerson. Vlfeat: An open and portable library of computer vision algorithms. In Proceedings of the international conference on Multimedia, pages 1469–1472. ACM, 2010. [36] S. Vicente, J. Carreira, L. Agapito, and J. Batista. Reconstructing pascal voc. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 41– 48. IEEE, 2014. [37] L. Wiskott and T. J. Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715–770, 2002. [38] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Computer vision and pattern recognition (CVPR), 2010 IEEE conference on, pages 3485–3492. IEEE, 2010. 12 Unsupervised Visual Representation Learning by Context Prediction Carl Doersch1,2 School of Computer Science Carnegie Mellon University 2 Alexei A. Efros2 Dept. of Electrical Engineering and Computer Science University of California, Berkeley Abstract Example: This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the RCNN framework [19] and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-theart performance among algorithms which use only Pascalprovided training set annotations. Question 1: Question 2: _? _? Figure 1. Our task for learning patch representations involves randomly sampling a patch (blue) and then one of eight possible neighbors (red). Can you guess the spatial configuration for the two pairs of patches? Note that the task is much easier once you have recognized the object! Answer key: Q1: Bottom right Q2: Top center arXiv:1505.05192v2 [cs.CV] 28 Sep 2015 1 Abhinav Gupta1 in the context (i.e., a few words before and/or after) given the vector. This converts an apparently unsupervised problem (finding a good similarity metric between words) into a “self-supervised” one: learning a function from a given word to the words surrounding it. Here the context prediction task is just a “pretext” to force the model to learn a good word embedding, which, in turn, has been shown to be useful in a number of real tasks, such as semantic word similarity [37]. Our paper aims to provide a similar “self-supervised” formulation for image data: a supervised task involving predicting the context for a patch. Our task is illustrated in Figures 1 and 2. We sample random pairs of patches in one of eight spatial configurations, and present each pair to a machine learner, providing no information about the patches’ original position within the image. The algorithm must then guess the position of one patch relative to the other. Our underlying hypothesis is that doing well on this task requires understanding scenes and objects, i.e. a good visual representation for this task will need to extract objects and their parts in order to reason about their relative spatial location. “Objects,” after all, consist of multiple parts that can be detected independently of one another, and which 1. Introduction Recently, new computer vision methods have leveraged large datasets of millions of labeled examples to learn rich, high-performance visual representations [29]. Yet efforts to scale these methods to truly Internet-scale datasets (i.e. hundreds of billions of images) are hampered by the sheer expense of the human annotation required. A natural way to address this difficulty would be to employ unsupervised learning, which aims to use data without any annotation. Unfortunately, despite several decades of sustained effort, unsupervised methods have not yet been shown to extract useful information from large collections of full-sized, real images. After all, without labels, it is not even clear what should be represented. How can one write an objective function to encourage a representation to capture, for example, objects, if none of the objects are labeled? Interestingly, in the text domain, context has proven to be a powerful source of automatic supervisory signal for learning representations [3, 38, 9, 37]. Given a large text corpus, the idea is to train a model that maps each word to a feature vector, such that it is easy to predict the words 1 occur in a specific spatial configuration (if there is no specific configuration of the parts, then it is “stuff” [1]). We present a ConvNet-based approach to learn a visual representation from this task. We demonstrate that the resulting visual representation is good for both object detection, providing a significant boost on PASCAL VOC 2007 compared to learning from scratch, as well as for unsupervised object discovery / visual data mining. This means, surprisingly, that our representation generalizes across images, despite being trained using an objective function that operates on a single image at a time. That is, instance-level supervision appears to improve performance on category-level tasks. 1 2 5 4 6 2. Related Work X=( One way to think of a good image representation is as the latent variables of an appropriate generative model. An ideal generative model of natural images would both generate images according to their natural distribution, and be concise in the sense that it would seek common causes for different images and share information between them. However, inferring the latent structure given an image is intractable for even relatively simple models. To deal with these computational issues, a number of works, such as the wake-sleep algorithm [23], contrastive divergence [22], deep Boltzmann machines [45], and variational Bayesian methods [28, 43] use sampling to perform approximate inference. Generative models have shown promising performance on smaller datasets such as handwritten digits [23, 22, 45, 28, 43], but none have proven effective for high-resolution natural images. Unsupervised representation learning can also be formulated as learning an embedding (i.e. a feature vector for each image) where images that are semantically similar are close, while semantically different ones are far apart. One way to build such a representation is to create a supervised “pretext” task such that an embedding which solves the task will also be useful for other real-world tasks. For example, denoising autoencoders [53, 4] use reconstruction from noisy data as a pretext task: the algorithm must connect images to other images with similar objects to tell the difference between noise and signal. Sparse autoencoders also use reconstruction as a pretext task, along with a sparsity penalty [39], and such autoencoders may be stacked to form a deep representation [32, 31]. (however, only [31] was successfully applied to full-sized images, requiring a million CPU hours to discover just three objects). We believe that current reconstruction-based algorithms struggle with lowlevel phenomena, like stochastic textures, making it hard to even measure whether a model is generating well. Another pretext task is “context prediction.” A strong tradition for this kind of task already exists in the text domain, where “skip-gram” [37] models have been shown to generate useful word representations. The idea is to train a 7 , 3 8 ); Y = 3 Figure 2. The algorithm receives two patches in one of these eight possible spatial arrangements, without any context, and must then classify which configuration was sampled. model (e.g. a deep network) to predict, from a single word, the n preceding and n succeeding words. In principle, similar reasoning could be applied in the image domain, a kind of visual “fill in the blank” task, but, again, one runs into the problem of determining whether the predictions themselves are correct [12], unless one cares about predicting only very low-level features [14, 30, 50]. To address this, [36] predicts the appearance of an image region by consensus voting of the transitive nearest neighbors of its surrounding regions. Our previous work [12] explicitly formulates a statistical test to determine whether the data is better explained by a prediction or by a low-level null hypothesis model. The key problem that these approaches must address is that predicting pixels is much harder than predicting words, due to the huge variety of pixels that can arise from the same semantic object. In the text domain, one interesting idea is to switch from a pure prediction task to a discrimination task [38, 9]. In this case, the pretext task is to discriminate true snippets of text from the same snippets where a word has been replaced at random. A direct extension of this to 2D might be to discriminate between real images vs. images where one patch has been replaced by a random patch from elsewhere in the dataset. However, such a task would be trivial, since discriminating low-level color statistics and lighting would be enough. To make the task harder and more high-level, in this paper, we instead classify between multiple possible configurations of patches sampled from the same image, which means they will share lighting and color statistics, as shown on Figure 2. Another line of work in unsupervised learning from images aims to discover object categories using hand-crafted features and various forms of clustering (e.g. [48, 44] learned a generative model over bags of visual words). Such representations lose shape information, and will readily dis2 cover clusters of, say, foliage. A few subsequent works have attempted to use representations more closely tied to shape [33, 40], but relied on contour extraction, which is difficult in complex images. Many other approaches [20, 27, 16] focus on defining similarity metrics which can be used in more standard clustering algorithms; [42], for instance, re-casts the problem as frequent itemset mining. Geometry may also be used to for verifying links between images [41, 6, 21], although this can fail for deformable objects. Video can provide another cue for representation learning. For most scenes, the identity of objects remains unchanged even as appearance changes with time. This kind of temporal coherence has a long history in visual learning literature [18, 56], and contemporaneous work shows strong improvements on modern detection datasets [54]. Finally, our work is related to a line of research on discriminative patch mining [13, 47, 26, 34, 49, 11], which has emphasized weak supervision as a means of object discovery. Like the current work, they emphasize the utility of learning representations of patches (i.e. object parts) before learning full objects and scenes, and argue that scene-level labels can serve as a pretext task. For example, [13] trains detectors to be sensitive to different geographic locales, but the actual goal is to discover specific elements of architectural style. fc9 (8) fc8 (4096) fc7 (4096) fc6 (4096) pool5 (3x3,256,2) conv5 (3x3,256,1) conv4 (3x3,384,1) conv3 (3x3,384,1) LRN2 pool2 (3x3,384,2) conv2 (5x5,384,2) LRN1 pool1 (3x3,96,2) conv1 (11x11,96,4) fc6 (4096) pool5 (3x3,256,2) conv5 (3x3,256,1) conv4 (3x3,384,1) conv3 (3x3,384,1) LRN2 pool2 (3x3,384,2) conv2 (5x5,384,2) LRN1 pool1 (3x3,96,2) conv1 (11x11,96,4) Patch 1 Patch 2 Figure 3. Our architecture for pair classification. Dotted lines indicate shared weights. ‘conv’ stands for a convolution layer, ‘fc’ stands for a fully-connected one, ‘pool’ is a max-pooling layer, and ‘LRN’ is a local response normalization layer. Numbers in parentheses are kernel size, number of outputs, and stride (fc layers have only a number of outputs). The LRN parameters follow [29]. All conv and fc layers are followed by ReLU nonlinearities, except fc9 which feeds into a softmax classifier. semantic reasoning for each patch separately. When designing the network, we followed AlexNet where possible. To obtain training examples given an image, we sample the first patch uniformly, without any reference to image content. Given the position of the first patch, we sample the second patch randomly from the eight possible neighboring locations as in Figure 2. 3. Learning Visual Context Prediction We aim to learn an image representation for our pretext task, i.e., predicting the relative position of patches within an image. We employ Convolutional Neural Networks (ConvNets), which are well known to learn complex image representations with minimal human feature design. Building a ConvNet that can predict a relative offset for a pair of patches is, in principle, straightforward: the network must feed the two input patches through several convolution layers, and produce an output that assigns a probability to each of the eight spatial configurations (Figure 2) that might have been sampled (i.e. a softmax output). Note, however, that we ultimately wish to learn a feature embedding for individual patches, such that patches which are visually similar (across different images) would be close in the embedding space. To achieve this, we use a late-fusion architecture shown in Figure 3: a pair of AlexNet-style architectures [29] that process each patch separately, until a depth analogous to fc6 in AlexNet, after which point the representations are fused. For the layers that process only one of the patches, weights are tied between both sides of the network, such that the same fc6-level embedding function is computed for both patches. Because there is limited capacity for joint reasoning—i.e., only two layers receive input from both patches—we expect the network to perform the bulk of the 3.1. Avoiding “trivial” solutions When designing a pretext task, care must be taken to ensure that the task forces the network to extract the desired information (high-level semantics, in our case), without taking “trivial” shortcuts. In our case, low-level cues like boundary patterns or textures continuing between patches could potentially serve as such a shortcut. Hence, for the relative prediction task, it was important to include a gap between patches (in our case, approximately half the patch width). Even with the gap, it is possible that long lines spanning neighboring patches could could give away the correct answer. Therefore, we also randomly jitter each patch location by up to 7 pixels (see Figure 2). However, even these precautions are not enough: we were surprised to find that, for some images, another trivial solution exists. We traced the problem to an unexpected culprit: chromatic aberration. Chromatic aberration arises from differences in the way the lens focuses light at different wavelengths. In some cameras, one color channel (commonly green) is shrunk toward the image center relative to the others [5, p. 76]. A ConvNet, it turns out, can learn to localize a patch relative to the lens itself (see Section 4.2) sim3 Input Random Initialization ImageNet AlexNet Ours Figure 4. Examples of patch clusters obtained by nearest neighbors. The query patch is shown on the far left. Matches are for three different features: fc6 features from a random initialization of our architecture, AlexNet fc7 after training on labeled ImageNet, and the fc6 features learned from our method. Queries were chosen from 1000 randomly-sampled patches. The top group is examples where our algorithm performs well; for the middle AlexNet outperforms our approach; and for the bottom all three features work well. ply by detecting the separation between green and magenta (red + blue). Once the network learns the absolute location on the lens, solving the relative location task becomes trivial. To deal with this problem, we experimented with two types of pre-processing. One is to shift green and magenta toward gray (‘projection’). Specifically, let a = [−1, 2, −1] (the ’green-magenta color axis’ in RGB space). We then define B = I − aT a/(aaT ), which is a matrix that subtracts the projection of a color onto the green-magenta color axis. We multiply every pixel value by B. An alternative approach is to randomly drop 2 of the 3 color channels from each patch (‘color dropping’), replacing the dropped colors with Gaussian noise (standard deviation ∼ 1/100 the standard deviation of the remaining channel). For qualitative results, we show the ‘color-dropping’ approach, but found both performed similarly; for the object detection results, we show both results. Implementation Details: We use Caffe [25], and train on the ImageNet [10] 2012 training set ( 1.3M images), using only the images and discarding the labels. First, we resize each image to between 150K and 450K total pixels, preserving the aspect-ratio. From these images, we sample patches at resolution 96-by-96. For computational efficiency, we only sample the patches from a grid like pattern, such that each sampled patch can participate in as many as 8 separate pairings. We allow a gap of 48 pixels between the sampled patches in the grid, but also jitter the location of each patch in the grid by −7 to 7 pixels in each direction. We preprocess patches by (1) mean subtraction (2) projecting or dropping colors (see above), and (3) randomly downsampling some patches to as little as 100 total pixels, and then upsampling it, to build robustness to pixelation. When ap4 Initial layout, with sampled patches in red Image layout is discarded We can recover image layout automatically Cannot recover layout with color removed Figure 5. We trained a network to predict the absolute (x, y) coordinates of randomly sampled patches. Far left: input image. Center left: extracted patches. Center right: the location the trained network predicts for each patch shown on the left. Far right: the same result after our color projection scheme. Note that the far right patches are shown after color projection; the operation’s effect is almost unnoticeable. plying simple SGD to train the network, we found that the network predictions would degenerate to a uniform prediction over the 8 categories, with all activations for fc6 and fc7 collapsing to 0. This meant that the optimization became permanently stuck in a saddle point where it ignored the input from the lower layers (which helped minimize the variance of the final output), and therefore that the net could not tune the lower-level features and escape the saddle point. Hence, Our final implementation employs batch normalization [24], which forces the network activations to vary across examples. We also find that high momentum values (e.g. .999) accelerated learning. For experiments, we use a ConvNet trained on a K40 GPU for approximately four weeks. fc8 (21) fc7 (4096) pool6 (3x3,1024,2) conv6b (1x1,1024,1) conv6 (3x3,4096,1) pool5 … Image (227x227) Figure 6. Our architecture for Pascal VOC detection. Layers from conv1 through pool5 are copied from our patch-based network (Figure 3). The new ’conv6’ layer is created by converting the fc6 layer into a convolution layer. Kernel sizes, output units, and stride are given in parentheses, as in Figure 3. tialization). As shown in Figure 4, the matches returned by our feature often capture the semantic information that we are after, matching AlexNet in terms of semantic content (in some cases, e.g. the car wheel, our matches capture pose better). Interestingly, in a few cases, random (untrained) ConvNet also does reasonably well. 4. Experiments 4.2. Aside: Learnability of Chromatic Aberration We first demonstrate the network has learned to associate semantically similar patches, using simple nearest-neighbor matching. We then apply the trained network in two domains. First, we use the model as “pre-training” for a standard vision task with only limited training data: specifically, we use the VOC 2007 object detection. Second, we evaluate visual data mining, where the goal is to start with an unlabeled image collection and discover object classes. Finally, we analyze the performance on the layout prediction “pretext task” to see how much is left to learn from this supervisory signal. We noticed in early nearest-neighbor experiments that some patches retrieved match patches from the same absolute location in the image, regardless of content, because those patches displayed similar aberration. To further demonstrate this phenomenon, we trained a network to predict the absolute (x, y) coordinates of patches sampled from ImageNet. While the overall accuracy of this regressor is not very high, it does surprisingly well for some images: for the top 10% of images, the average (root-mean-square) error is .255, while chance performance (always predicting the image center) yields a RMSE of .371. Figure 5 shows one such result. Applying the proposed “projection” scheme increases the error on the top 10% of images to .321. 4.1. Nearest Neighbors Recall our intuition that training should assign similar representations to semantically similar patches. In this section, our goal is to understand which patches our network considers similar. We begin by sampling random 96x96 patches, which we represent using fc6 features (i.e. we remove fc7 and higher shown in Figure 3, and use only one of the two stacks). We find nearest neighbors using normalized correlation of these features. Results for some patches (selected out of 1000 random queries) are shown in Figure 4. For comparison, we repeated the experiment using fc7 features from AlexNet trained on ImageNet (obtained by upsampling the patches), and using fc6 features from our architecture but without any training (random weights ini- 4.3. Object Detection Previous work on the Pascal VOC challenge [15] has shown that pre-training on ImageNet (i.e., training a ConvNet to solve the ImageNet challenge) and then “finetuning” the network (i.e. re-training the ImageNet model for PASCAL data) provides a substantial boost over training on the Pascal training set alone [19, 2]. However, as far as we are aware, no works have shown that unsupervised pretraining on images can provide such a performance boost, no matter how much data is used. Since we are already using a ConvNet, we adopt the cur5 VOC-2007 Test DPM-v5[17] [8] w/o context Regionlets[55] Scratch-R-CNN[2] Scratch-Ours Ours-projection Ours-color-dropping Ours-Yahoo100m Ours-VGG aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mAP 16.1 25.4 24.0 23.7 24.3 27.7 28.5 27.8 27.3 18.7 20.1 20.3 18.1 24.4 26.3 23.9 54.3 47.3 55.5 52.5 50.6 58.5 56.1 57.4 58.2 56.9 68.7 64.8 65.9 68.5 70.4 69.8 23.0 42.1 42.6 32.9 29.2 41.2 44.8 35.6 20.0 16.6 19.2 20.4 19.5 26.3 24.6 23.7 12.7 27.7 26.6 29.9 27.6 37.3 35.1 29.5 58.1 47.9 57.0 49.0 46.5 55.7 52.2 52.9 48.2 51.5 54.5 60.4 59.4 62.5 60.2 62.0 43.2 29.9 43.4 47.5 46.5 49.4 50.0 48.7 12.0 20.0 16.4 28.0 25.6 29.0 28.1 28.4 21.1 41.1 36.6 42.3 42.4 47.5 46.7 45.1 36.1 36.4 37.7 28.6 23.5 28.4 42.6 33.6 43.5 53.2 52.3 50.0 50.6 56.8 58.6 55.5 33.7 38.5 41.7 40.7 39.8 45.7 46.3 44.2 63.6 64.4 42.0 42.9 18.9 67.9 69.5 65.9 28.2 48.1 58.4 58.5 66.2 64.9 54.1 26.1 43.9 55.9 69.8 50.9 53.0 32.0 62.6 71.0 60.7 32.7 58.5 46.5 56.1 60.6 66.8 54.2 31.5 52.8 48.9 57.9 64.7 54.2 33.2 52.6 54.2 49.9 52.6 58.4 60.5 56.2 60.3 52.6 52.0 60.6 60.5 62.8 66.5 63.9 ImageNet-R-CNN[19] 64.2 69.7 10.2 19.2 20.3 24.7 23.8 33.5 29.6 29.8 50 41.9 24.1 41.4 44.2 43.5 43.5 49.5 45.5 47.4 26.7 41.9 49.1 34.2 35.2 42.6 45.4 43.0 46.0 48.6 59.4 51.2 50.0 54.7 54.8 49.0 Table 1. Results on VOC-2007. R-CNN performance with our unsupervised pre-training is 5% MAP better than training from scratch, but still 8% below pre-training with ImageNet label supervision. rent state-of-the-art R-CNN pipeline [19]. R-CNN works on object proposals that have been resized to 227x227. Our algorithm, however, is aimed at 96x96 patches. We find that downsampling the proposals to 96x96 loses too much detail. Instead, we adopt the architecture shown in Figure 6. As above, we use only one stack from Figure 3. Second, we resize the convolution layers to operate on inputs of 227x227. This results in a pool5 that is 7x7 spatially, so we must convert the previous fc6 layer into a convolution layer (which we call conv6) following [35]. Note our conv6 layer has 4096 channels, where each unit connects to a 3x3 region of pool5. A conv layer with 4096 channels would be quite expensive to connect directly to a 4096-dimensional fullyconnected layer. Hence, we add another layer after conv6 (called conv6b), using a 1x1 kernel, which reduces the dimensionality to 1024 channels (and adds a nonlinearity). Finally, we feed the outputs through a pooling layer to a fully connected layer (fc7) which in turn connects to a final fc8 layer which feeds into the softmax. We fine-tune this network according to the procedure described in [19] (conv6b, fc7, and fc8 start with random weights), and use fc7 as the final representation. We do not use boundingbox regression, and take the appropriate results from [19] and [2]. Table 1 shows our results. Our architecture trained from scratch (random initialization) performs slightly worse than AlexNet trained from scratch. However, our pre-training makes up for this, boosting the from-scratch number by 6% MAP, and outperforms an AlexNet-style model trained from scratch on Pascal by over 5%. This puts us about 8% behind the performance of R-CNN pre-trained with ImageNet labels [19]. This is the best result we are aware of on VOC 2007 without using labels outside the dataset. We ran additional baselines initialized with batch normalization, but found they performed worse than the ones shown. To understand the effect of various dataset biases [52], we also performed a preliminary experiment pre-training on a randomly-selected 2M subset of the Yahoo/Flickr 100million Dataset [51], which was collected entirely automatically. The performance after fine-tuning is slightly worse than Imagenet, but there is still a considerable boost over the from-scratch model. We also performed a preliminary ex- periment with a VGG-style [46] (16-layer) network, shown as “Ours-VGG” in Table 1. 4.4. Visual Data Mining Visual data mining [41, 13, 47, 42], or unsupervised object discovery [48, 44, 20], aims to use a large image collection to discover image fragments which happen to depict the same semantic objects. Applications include dataset visualization, content-based retrieval, and tasks that require relating visual data to other unstructured information (e.g. GPS coordinates [13]). For automatic data mining, our approach from section 4.1 is inadequate: although object patches match to similar objects, textures match just as readily to similar textures. Suppose, however, that we sampled two non-overlapping patches from the same object. Not only would the nearest neighbor lists for both patches share many images, but within those images, the nearest neighbors would be in roughly the same spatial configuration. For texture regions, on the other hand, the spatial configurations of the neighbors would be random, because the texture has no global layout. To implement this, we first sample a constellation of four adjacent patches from an image (we use four to reduce the likelihood of a matching spatial arrangement happening by chance). We find the top 100 images which have the strongest matches for all four patches, ignoring spatial layout. We then use a type of geometric verification [7] to filter away the images where the four matches are not geometrically consistent. Because our features are more semantically-tuned, we can use a much weaker type of geometric verification than [7]. Finally, we rank the different constellations by counting the number of times the top 100 matches geometrically verify. Implementation Details: To compute whether a set of four matched patches geometrically verifies, we first compute the best-fitting square S to the patch centers (via leastsquares), while constraining that side of S be between 2/3 and 4/3 of the average side of the patches. We then compute the squared error of the patch centers relative to S (normalized by dividing the sum-of-squared-errors by the square of the side of S). The patch is geometrically verified if this normalized squared error is less than 1. When sampling 6 1 88 4 121 7 131 12 142 25 179 29 187 30 229 229 35 232 232 46 240 70 256 71 351 73 464 Figure 7. Object clusters discovered by our algorithm. The number beside each cluster indicates its ranking, determined by the fraction of the top matches that geometrically verified. For all clusters, we show the raw top 7 matches that verified geometrically. The full ranking is available on our project webpage. patches do not use any of the data augmentation preprocessing steps (e.g. downsampling). We use the color-dropping version of our network. As in [12], we often re-discover the same object multiple times with different viewpoints, which accounts for most of the gaps between ranks in Figure 7. The main disadvantages of our algorithm relative to [12] are 1) some loss of purity, and 2) that we cannot currently determine an object mask automatically (although one could imagine dynamically adding more sub-patches to each proposed object). To ensure that our algorithm has not simply learned an object-centric representation due to the various biases [52] in ImageNet, we also applied our algorithm to 15,000 Street View images from Paris (following [13]). The results in Figure 8 show that our representation captures scene layout and architectural elements. For this experiment, to rank clusters, we use the de-duplication procedure originally proposed in [13]. We applied the described mining algorithm to Pascal VOC 2011, with no pre-filtering of images and no additional labels. We show some of the resulting patch clusters in Figure 7. The results are visually comparable to our previous work [12], although we discover a few objects that were not found in [12], such as monitors, birds, torsos, and plates of food. The discovery of birds and torsos—which are notoriously deformable—provides further evidence for the invariances our algorithm has learned. We believe we have covered all objects discovered in [12], with the exception of (1) trusses and (2) railroad tracks without trains (though we do discover them with trains). For some objects like dogs, we discover more variety and rank the best ones higher. Furthermore, many of the clusters shown in [12] depict gratings (14 out of the top 100), whereas none of ours do (though two of our top hundred depict diffuse gradients). 4.4.1 Quantitative Results As part of the qualitative evaluation, we applied our algorithm to the subset of Pascal VOC 2007 selected in [47]: specifically, those containing at least one instance of bus, 7 1.0 1: Purity-Coverage for Proposed Objects 0.9 0.8 4: 0.7 Purity 0.6 5: 0.5 0.4 Visual Words .63 (.37) Russel et al. .66 (.38) HOG Kmeans .70 (.40) Singh et al. .83 (.47) Doersch et al. .83 (.48) Our Approach .87 (.48) 0.3 6: 0.2 0.1 13: 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Coverage Figure 9. Purity vs coverage for objects discovered on a subset of Pascal VOC 2007. The numbers in the legend indicate area under the curve (AUC). In parentheses is the AUC up to a coverage of .5. 18: briefly analyze classification performance on pretext task itself. We sampled 500 random images from Pascal VOC 2007, sampled 256 pairs of patches from each, and classified them into the eight relative-position categories from Figure 2. This gave an accuracy of 38.4%, where chance performance is 12.5%, suggesting that the pretext task is quite hard (indeed, human performance on the task is similar). To measure possible overfitting, we also ran the same experiment on ImageNet, which is the dataset we used for training. The network was 39.5% accurate on the training set, and 40.3% accurate on the validation set (which the network never saw during training), suggesting that little overfitting has occurred. One possible reason why the pretext task is so difficult is because, for a large fraction of patches within each image, the task is almost impossible. Might the task be easiest for image regions corresponding to objects? To test this hypothesis, we repeated our experiment using only patches sampled from within Pascal object ground-truth bounding boxes. We select only those boxes that are at least 240 pixels on each side, and which are not labeled as truncated, occluded, or difficult. Surprisingly, this gave essentially the same accuracy of 39.2%, and a similar experiment only on cars yielded 45.6% accuracy. So, while our algorithm is sensitive to objects, it is almost as sensitive to the layout of the rest of the image. 42: 53: Figure 8. Clusters discovered and automatically ranked via our algorithm (§ 4.4) from the Paris Street View dataset. dining table, motorbike, horse, sofa, or train, and evaluate via a purity coverage curve following [12]. We select 1000 sets of 10 images each for evaluation. The evaluation then sorts the sets by purity: the fraction of images in the cluster containing the same category. We generate the curve by walking down the ranking. For each point on the curve, we plot average purity of all sets up to a given point in the ranking against coverage: the fraction of images in the dataset that are contained in at least one of the sets up to that point. As shown in Figure 9, we have gained substantially in terms of coverage, suggesting increased invariance for our learned feature. However, we have also lost some highly-pure clusters compared to [12]—which is not very surprising considering that our validation procedure is considerably simpler. Implementation Details: We initialize 16,384 clusters by sampling patches, mining nearest neighbors, and geometric verification ranking as described above. The resulting clusters are highly redundant. The cluster selection procedure of [12] relies on a likelihood ratio score that is calibrated across clusters, which is not available to us. To select clusters, we first select the top 10 geometrically-verified neighbors for each cluster. Then we iteratively select the highest-ranked cluster that contributes at least one image to our coverage score. When we run out of images that aren’t included in the coverage score, we choose clusters to cover each image at least twice, and then three times, and so on. Acknowledgements We thank Xiaolong Wang and Pulkit Agrawal for help with baselines, Berkeley and CMU vision group members for many fruitful discussions, and Jitendra Malik for putting gelato on the line. This work was partially supported by Google Graduate Fellowship to CD, ONR MURI N000141010934, Intel research grant, an NVidia hardware grant, and an Amazon Web Services grant. References [1] E. H. Adelson. On seeing stuff: the perception of materials by humans and machines. In Photonics West 2001-Electronic Imaging, 2001. 2 [2] P. Agrawal, R. Girshick, and J. Malik. Analyzing the performance of multilayer neural networks for object recognition. In ECCV. 2014. 4.5. Accuracy on the Relative Prediction Task Task Can we improve the representation by further training on our relative prediction pretext task? To find out, we 8 5, 6 [3] R. K. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 2005. 1 [4] Y. Bengio, E. Thibodeau-Laufer, G. Alain, and J. Yosinski. Deep generative stochastic networks trainable by backprop. ICML, 2014. 2 [5] D. Brewster and A. D. Bache. Treatise on optics. Blanchard and Lea, 1854. 3 [6] O. Chum, M. Perdoch, and J. Matas. Geometric min-hashing: Finding a (thick) needle in a haystack. In CVPR, 2009. 3 [7] O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman. Total recall: Automatic query expansion with a generative feature model for object retrieval. In ICCV, 2007. 6 [8] R. G. Cinbis, J. Verbeek, and C. Schmid. Segmentation driven object detection with Fisher vectors. In ICCV, 2013. 6 [9] R. Collobert and J. Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, 2008. 1, 2 [10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 4 [11] C. Doersch, A. Gupta, and A. A. Efros. Mid-level visual element discovery as discriminative mode seeking. In NIPS, 2013. 3 [12] C. Doersch, A. Gupta, and A. A. Efros. Context as supervisory signal: Discovering objects with predictable context. In ECCV. 2014. 2, 7, 8 [13] C. Doersch, S. Singh, A. Gupta, J. Sivic, and A. A. Efros. What makes Paris look like Paris? SIGGRAPH, 2012. 3, 6, 7 [14] J. Domke, A. Karapurkar, and Y. Aloimonos. Who killed the directed model? In CVPR, 2008. 2 [15] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010. 5 [16] A. Faktor and M. Irani. clustering by composition–unsupervised discovery of image categories. In ECCV. 2012. 3 [17] P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. PAMI, 2010. 6 [18] P. Földiák. Learning invariance from transformation sequences. Neural Computation, 1991. 3 [19] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 1, 5, 6 [20] K. Grauman and T. Darrell. Unsupervised learning of categories from sets of partially matching image features. In CVPR, 2006. 3, 6 [21] K. Heath, N. Gelfand, M. Ovsjanikov, M. Aanjaneya, and L. J. Guibas. Image webs: Computing and exploiting connectivity in image collections. In CVPR, 2010. 3 [22] G. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural computation, 2006. 2 [23] G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal. The “wakesleep” algorithm for unsupervised neural networks. Proceedings. IEEE, 1995. 2 [24] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. 5 [25] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM-MM, 2014. 4 [26] M. Juneja, A. Vedaldi, C. V. Jawahar, and A. Zisserman. Blocks that shout: Distinctive parts for scene classification. In CVPR, 2013. 3 [27] G. Kim, C. Faloutsos, and M. Hebert. Unsupervised modeling of object categories using link analysis techniques. In CVPR, 2008. 3 [28] D. P. Kingma and M. Welling. Auto-encoding variational bayes. 2014. 2 [29] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 1, 3 [30] H. Larochelle and I. Murray. The neural autoregressive distribution estimator. In AISTATS, 2011. 2 [31] Q. V. Le. Building high-level features using large scale unsupervised learning. In ICASSP, 2013. 2 [32] H. Lee, A. Battle, R. Raina, and A. Y. Ng. Efficient sparse coding algorithms. In NIPS, 2006. 2 [33] Y. J. Lee and K. Grauman. Foreground focus: Unsupervised learning from partially matching images. IJCV, 2009. 3 [34] Q. Li, J. Wu, and Z. Tu. Harvesting mid-level visual concepts from large-scale internet images. In CVPR, 2013. 3 [35] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. arXiv preprint arXiv:1411.4038, 2014. 6 [36] T. Malisiewicz and A. Efros. Beyond categories: The visual memex model for reasoning about object relationships. In NIPS, 2009. 2 [37] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013. 1, 2 [38] D. Okanohara and J. Tsujii. A discriminative language model with pseudo-negative samples. In ACL, 2007. 1, 2 [39] B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 1996. 2 [40] N. Payet and S. Todorovic. From a set of shapes to object discovery. In ECCV. 2010. 3 [41] T. Quack, B. Leibe, and L. Van Gool. World-scale mining of objects and events from community photo collections. In CIVR, 2008. 3, 6 [42] K. Rematas, B. Fernando, F. Dellaert, and T. Tuytelaars. Dataset fingerprints: Exploring image collections through data mining. In CVPR, 2015. 3, 6 [43] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. ICML, 2014. 2 [44] B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman. Using multiple segmentations to discover objects and their extent in image collections. In CVPR, 2006. 2, 6 [45] R. Salakhutdinov and G. E. Hinton. Deep boltzmann machines. In ICAIS, 2009. 2 [46] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 2014. 6 [47] S. Singh, A. Gupta, and A. A. Efros. Unsupervised discovery of mid-level discriminative patches. In ECCV, 2012. 3, 6, 7 [48] J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, and W. T. Freeman. Discovering objects and their location in images. In ICCV, 2005. 2, 6 [49] J. Sun and J. Ponce. Learning discriminative part detectors for image classification and cosegmentation. In ICCV, 2013. 3 [50] L. Theis and M. Bethge. Generative image modeling using spatial lstms. In NIPS, 2015. 2 [51] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. The new data and new challenges in multimedia research. arXiv preprint arXiv:1503.01817, 2015. 6 [52] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR, 2011. 6, 7 [53] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008. 2 [54] X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015. 3 [55] X. Wang, M. Yang, S. Zhu, and Y. Lin. Regionlets for generic object detection. In ICCV, 2013. 6 [56] L. Wiskott and T. J. Sejnowski. Slow feature analysis:unsupervised learning of invariances. Neural Computation, 2002. 3 9 PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization Alex Kendall Matthew Grimes University of Cambridge Roberto Cipolla arXiv:1505.07427v3 [cs.CV] 23 Nov 2015 agk34, mkg30, rc10001 @cam.ac.uk King’s College Old Hospital Shop Façade St Mary’s Church Figure 1: PoseNet: Convolutional neural network monocular camera relocalization. Relocalization results for an input image (top), the predicted camera pose of a visual reconstruction (middle), shown again overlaid in red on the original image (bottom). Our system relocalizes to within approximately 2m and 3◦ for large outdoor scenes spanning 50, 000m2 . For an online demonstration, please see our project webpage: mi.eng.cam.ac.uk/projects/relocalisation/ Abstract We present a robust and real-time monocular six degree of freedom relocalization system. Our system trains a convolutional neural network to regress the 6-DOF camera pose from a single RGB image in an end-to-end manner with no need of additional engineering or graph optimisation. The algorithm can operate indoors and outdoors in real time, taking 5ms per frame to compute. It obtains approximately 2m and 3◦ accuracy for large scale outdoor scenes and 0.5m and 5◦ accuracy indoors. This is achieved using an efficient 23 layer deep convnet, demonstrating that convnets can be used to solve complicated out of image plane regression problems. This was made possible by leveraging transfer learning from large scale classification data. We show that the PoseNet localizes from high level features and is robust to difficult lighting, motion blur and different camera intrinsics where point based SIFT registration fails. Furthermore we show how the pose feature that is produced generalizes to other scenes allowing us to regress pose with only a few dozen training examples. 1. Introduction Inferring where you are, or localization, is crucial for mobile robotics, navigation and augmented reality. This paper addresses the lost or kidnapped robot problem by introducing a novel relocalization algorithm. Our proposed system, PoseNet, takes a single 224x224 RGB image and regresses the camera’s 6-DoF pose relative to a scene. Fig. 1 demonstrates some examples. The algorithm is simple in the fact that it consists of a convolutional neural network (convnet) trained end-to-end to regress the camera’s orientation and position. It operates in real time, taking 5ms to run, and obtains approximately 2m and 3 degrees accuracy for large scale outdoor scenes (covering a ground area of up to 50, 000m2 ). Our main contribution is the deep convolutional neural network camera pose regressor. We introduce two novel techniques to achieve this. We leverage transfer learning from recognition to relocalization with very large scale classification datasets. Additionally we use structure from motion to automatically generate training labels (camera poses) from a video of the scene. This reduces the human labor in creating labeled video datasets to just recording the video. Our second main contribution is towards understanding the representations that this convnet generates. We show that the system learns to compute feature vectors which are easily mapped to pose, and which also generalize to unseen scenes with a few additional training samples. Appearance-based relocalization has had success [4, 23] in coarsely locating the camera among a limited, discretized set of place labels, leaving the pose estimation to a separate system. This paper presents a means of computing continuous pose directly from appearance. The scene may include multiple objects and need not be viewed under consistent conditions. For example the scene may include dynamic objects like people and cars or experience changing weather conditions. Simultaneous localization and mapping (SLAM) is a traditional solution to this problem. We introduce a new framework for localization which removes several issues faced by typical SLAM pipelines, such as the need to store densely spaced keyframes, the need to maintain separate mechanisms for appearance-based localization and landmark-based pose estimation, and a need to establish frame-to-frame feature correspondence. We do this by mapping monocular images to a high-dimensional representation that is robust to nuisance variables. We empirically show that this representation is a smoothly varying injective (one-to-one) function of pose, allowing us to regress pose directly from the image without need of tracking. Training convolutional networks is usually dependent on very large labeled image datasets, which are costly to assemble. Examples include the ImageNet [5] and Places [29] datasets, with 14 million and 7 million hand-labeled images, respectively. We employ two techniques to overcome this limitation: • an automated method of labeling data using structure from motion to generate large regression datasets of camera pose • transfer learning which trains a pose regressor, pretrained as a classifier, on immense image recognition datasets. This converges to a lower error in less time, even with a very sparse training set, as compared to training from scratch. 2. Related work There are generally two approaches to localization: metric and appearance-based. Metric SLAM localizes a mobile robot by focusing on creating a sparse [13, 11] or dense [16, 7] map of the environment. Metric SLAM estimates the camera’s continuous pose, given a good initial pose estimate. Appearance-based localization provides this coarse estimate by classifying the scene among a limited number of discrete locations. Scalable appearance-based localiz- ers have been proposed such as [4] which uses SIFT features [15] in a bag of words approach to probabilistically recognize previously viewed scenery. Convnets have also been used to classify a scene into one of several location labels [23]. Our approach combines the strengths of these approaches: it does not need an initial pose estimate, and produces a continuous pose. Note we do not build a map, rather we train a neural network, whose size, unlike a map, does not require memory linearly proportional to the size of the scene (see fig. 13). Our work most closely follows from the Scene Coordinate Regression Forests for relocalization proposed in [20]. This algorithm uses depth images to create scene coordinate labels which map each pixel from camera coordinates to global scene coordinates. This was then used to train a regression forest to regress these labels and localize the camera. However, unlike our approach, this algorithm is limited to RGB-D images to generate the scene coordinate label, in practice constraining its use to indoor scenes. Previous research such as [27, 14, 9, 3] has also used SIFT-like point based features to match and localize from landmarks. However these methods require a large database of features and efficient retrieval methods. A method which uses these point features is structure from motion (SfM) [28, 1, 22] which we use here as an offline tool to automatically label video frames with camera pose. We use [8] to generate a dense visualisation of our relocalization results. Despite their ability in classifying spatio-temporal data, convolutional neural networks are only just beginning to be used for regression. They have advanced the state of the art in object detection [24] and human pose regression [25]. However these have limited their regression targets to lie in the 2-D image plane. Here we demonstrate regressing the full 6-DOF camera pose transform including depth and out-of-plane rotation. Furthermore, we show we are able to learn regression as opposed to being a very fine resolution classifier. It has been shown that convnet representations trained on classification problems generalize well to other tasks [18, 17, 2, 6]. We show that you can apply these representations of classification to 6-DOF regression problems. Using these pre-learned representations allows convnets to be used on smaller datasets without overfitting. 3. Model for deep regression of camera pose In this section we describe the convolutional neural network (convnet) we train to estimate camera pose directly from a monocular image, I. Our network outputs a pose vector p, given by a 3D camera position x and orientation represented by quaternion q: p = [x, q] (1) Pose p is defined relative to an arbitrary global reference frame. We chose quaternions as our orientation representation, because arbitrary 4-D values are easily mapped to legitimate rotations by normalizing them to unit length. This is a simpler process than the orthonormalization required of rotation matrices. 3.1. Simultaneously learning location and orientation To regress pose, we train the convnet on Euclidean loss using stochastic gradient descent with the following objective loss function: loss(I) = kx̂ − xk2 + β q̂ − q kqk Figure 2: Relative performance of position and orientation regression on a single convnet with a range of scale factors for an indoor scene, Chess. This demonstrates that learning with the optimum scale factor leads to the convnet uncovering a more accurate pose function. (2) 2 Where β is a scale factor chosen to keep the expected value of position and orientation errors to be approximately equal. The set of rotations lives on the unit sphere in quaternion space. However the Euclidean loss function makes no effort to keep q on the unit sphere. We find, however, that during training, q becomes close enough to q̂ such that the distinction between spherical distance and Euclidean distance becomes insignificant. For simplicity, and to avoid hampering the optimization with unnecessary constraints, we chose to omit the spherical constraint. We found that training individual networks to regress position and orientation separately performed poorly compared to when they were trained with full 6-DOF pose labels (fig. 2). With just position, or just orientation information, the convnet was not as effectively able to determine the function representing camera pose. We also experimented with branching the network lower down into two separate components to regress position and orientation. However, we found that it too was less effective, for similar reasons: separating into distinct position and orientation regressors denies each the information necessary to factor out orientation from position, or vice versa. In our loss function (2) a balance β must be struck between the orientation and translation penalties (fig. 2). They are highly coupled as they are regressed from the same model weights. We observed that the optimal β was given by the ratio between expected error of position and orientation at the end of training, not the beginning. We found β to be greater for outdoor scenes as position errors tended to be relatively greater. Following this intuition we fine tuned β using grid search. For the indoor scenes it was between 120 to 750 and outdoor scenes between 250 to 2000. We found it was important to randomly initialize the final position regressor layer so that the norm of the weights corresponding to each position dimension was proportional to that dimension’s spatial extent. Classification problems have a training example for every category. This is not possible for regression as the output is continuous and infinite. Furthermore, other convnets that have been used for regression operate off very large datasets [25, 19]. For localization regression to work off limited data we leverage the powerful representations learned off these large classification datasets by pretraining the weights on these datasets. 3.2. Architecture For the experiments in this paper we use a state of the art deep neural network architecture for classification, GoogLeNet [24], as a basis for developing our pose regression network. GoogLeNet is a 22 layer convolutional network with six ‘inception modules’ and two additional intermediate classifiers which are discarded at test time. Our model is a slightly modified version of GoogLeNet with 23 layers (counting only the layers with trainable parameters). We modified GoogLeNet as follows: • Replace all three softmax classifiers with affine regressors. The softmax layers were removed and each final fully connected layer was modified to output a pose vector of 7-dimensions representing position (3) and orientation (4). • Insert another fully connected layer before the final regressor of feature size 2048. This was to form a localization feature vector which may then be explored for generalisation. • At test time we also normalize the quaternion orientation vector to unit length. We rescaled the input image so that the smallest dimension was 256 pixels before cropping to the 224x224 pixel input to the GoogLeNet convnet. The convnet was trained on random crops (which do not affect the camera pose). At test time we evaluate it with both a single center crop and also densely with 128 uniformly spaced crops of the input image, averaging the resulting pose vectors. With parallel GPU processing, this results in a computational time increase from 5ms to 95ms per image. Figure 3: Magnified view of a sequence of training (green) and testing (blue) cameras for King’s College. We show the predicted camera pose in red for each testing frame. The images show the test image (top), the predicted view from our convnet overlaid in red on the input image (middle) and the nearest neighbour training image overlaid in red on the input image (bottom). This shows our system can interpolate camera pose effectively in space between training frames. We experimented with rescaling the original image to different sizes before cropping for training and testing. Scaling up the input is equivalent to cropping the input before downsampling to 256 pixels on one side. This increases the spatial resolution of the input pixels. We found that this does not increase the localization performance, indicating that context and field of view is more important than resolution for relocalization. The PoseNet model was implemented using the Caffe library [10]. It was trained using stochastic gradient descent with a base learning rate of 10− 5, reduced by 90% every 80 epochs and with momentum of 0.9. Using one half of a dual-GPU card (NVidia Titan Black), training took an hour using a batch size of 75. For reasons of time, we did not explore multi-GPU training, although it is reasonable to expect better results from using double the throughput and memory. We subtracted a separate image mean for each scene as we found this to improve experimental performance. 4. Dataset Deep learning performs extremely well on large datasets, however producing these datasets is often expensive or very labour intensive. We overcome this by leveraging structure from motion to autonomously generate training labels (camera poses). This reduces the human labour to just recording the video of each scene. For this paper we release an outdoor urban localization dataset, Cambridge Landmarks1 , with 5 scenes. This novel dataset provides data to train and test pose regression algorithms in a large scale outdoor urban setting. A bird’s eye view of the camera poses is shown in fig. 4 and further de1 To download the dataset please see our project webpage: mi.eng.cam.ac.uk/projects/relocalisation/ tails can be found in table 6. Significant urban clutter such as pedestrians and vehicles were present and data was collected from many different points in time representing different lighting and weather conditions. Train and test images are taken from distinct walking paths and not sampled from the same trajectory making the regression challenging (see fig. 3). We release this dataset for public use and hope to add scenes to this dataset as this project progresses. The dataset was generated using structure from motion techniques [28] which we use as ground truth measurements for this paper. A Google LG Nexus 5 smartphone was used by a pedestrian to take high definition video around each scene. This video was subsampled in time at 2Hz to generate images to input to the SfM pipeline. There is a spacing of about 1m between each camera position. To test on indoor scenes we use the publically available 7 Scenes dataset [20], with scenes shown in fig. 5. This dataset contains significant variation in camera height and was designed for RGB-D relocalization. It is extremely challenging for purely visual relocalization using SIFT-like features, as it contains many ambiguous textureless features. 5. Experiments We show that PoseNet is able to effectively localize across both the indoor 7 Scenes dataset and outdoor Cambridge Landmarks dataset in table 6. To validate that the convnet is regressing pose beyond that of the training examples we show the performance for finding the nearest neighbour representation in the training data from the feature vector produced by the localization convnet. As our performance exceeds this we conclude that the convnet is successfully able to regress pose beyond training examples (see fig. 3). We also compare our algorithm to the RGB-D SCoRe Forest algorithm [20]. Fig. 7 shows cumulative histograms of localization error for two indoor and two outdoor scenes. We note that although the SCoRe forest is generally more accurate, it requires depth information, and uses higher-resolution imagery. The indoor dataset contains many ambiguous and textureless features which make relocalization without this depth modality extremely difficult. We note our method often localizes the most difficult testing frames, above the 95th percentile, more accurately than SCoRe across all the scenes. We also observe that dense cropping only gives a modest improvement in performance. It is most important in scenes with significant clutter like pedestrians and cars, for example King’s College, Shop Façade and St Mary’s Church. We explored the robustness of this method beyond what was tested in the dataset with additional images from dusk, rain, fog, night and with motion blur and different cameras with unknown intrinsics. Fig. 8 shows the convnet gener- King’s College Street Old Hospital Shop Façade St Mary’s Church Figure 4: Map of dataset showing training frames (green), testing frames (blue) and their predicted camera pose (red). The testing sequences are distinct trajectories from the training sequences and each scene covers a very large spatial extent. Figure 5: 7 Scenes dataset example images from left to right; Chess, Fire, Heads, Office, Pumpkin, Red Kitchen and Stairs. # Frames Train Test 1220 343 3015 2923 895 182 231 103 1487 530 4000 2000 2000 2000 1000 1000 6000 4000 4000 2000 7000 5000 2000 1000 Scene King’s College Street Old Hospital Shop Façade St Mary’s Church Chess Fire Heads Office Pumpkin Red Kitchen Stairs Spatial Extent (m) 140 x 40m 500 x 100m 50 x 40m 35 x 25m 80 x 60m 3 x 2 x 1m 2.5 x 1 x 1m 2 x 0.5 x 1m 2.5 x 2 x 1.5m 2.5 x 2 x 1m 4 x 3 x 1.5m 2.5 x 2 x 1.5m SCoRe Forest (Uses RGB-D) N/A N/A N/A N/A N/A 0.03m, 0.66◦ 0.05m, 1.50◦ 0.06m, 5.50◦ 0.04m, 0.78◦ 0.04m, 0.68◦ 0.04m, 0.76◦ 0.32m, 1.32◦ Dist. to Conv. Nearest Neighbour 3.34m, 2.96◦ 1.95m, 4.51◦ 5.38m, 4.51◦ 2.10m, 5.20◦ 4.48m, 5.65◦ 0.41m, 5.60◦ 0.54m, 7.77◦ 0.28m, 7.00◦ 0.49m, 6.02◦ 0.58m, 6.08◦ 0.58m, 5.65◦ 0.56m, 7.71◦ PoseNet 1.92m, 2.70◦ 3.67m, 3.25◦ 2.31m, 2.69◦ 1.46m, 4.04◦ 2.65m, 4.24◦ 0.32m, 4.06◦ 0.47m, 7.33◦ 0.29m, 6.00◦ 0.48m, 3.84◦ 0.47m, 4.21◦ 0.59m, 4.32◦ 0.47m, 6.93◦ Dense PoseNet 1.66m, 2.43◦ 2.96m, 3.00◦ 2.62m, 2.45◦ 1.41m, 3.59◦ 2.45m, 3.98◦ 0.32m, 3.30◦ 0.47m, 7.02◦ 0.30m, 6.09◦ 0.48m, 3.62◦ 0.49m, 4.06◦ 0.58m, 4.17◦ 0.48m, 6.54◦ Figure 6: Dataset details and results. We show median performance for PoseNet on all scenes, evaluated on a single 224x224 center crop and 128 uniformly separated dense crops. For comparison we plot the results from SCoRe Forest [20] which uses depth, therefore fails on outdoor scenes. This system regresses pixel-wise world coordinates of the input image at much larger resolution. This requires a dense depth map for training and an extra RANSAC step to determine the camera’s pose. Additionally, we compare to matching the nearest neighbour feature vector representation from PoseNet. This demonstrates our regression PoseNet performs better than a classifier. 1 1 1 1 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0 0 1 2 3 4 5 6 7 8 0 0 5 Positional error (m) 10 15 0 1 0.8 0.8 0.8 0.6 0.6 0.6 0 0.4 Nearest Neighbour CNN PoseNet Dense PoseNet 0 1 2 3 4 5 6 Angular error (degrees) (a) King’s College 7 8 0 0 2 4 6 8 10 12 1.5 0 0 0.5 14 16 Angular error (degrees) (b) St Mary’s Church 18 0 1.5 0.8 0.6 SCORE Forest PoseNet Dense PoseNet Nearest Neighbour CNN 0.2 20 1 Positional error (m) 1 0.4 Nearest Neighbour CNN PoseNet Dense PoseNet 0.2 1 Positional error (m) 1 0.2 0.5 Positional error (m) 1 0.4 0.2 0 0 5 10 15 20 Angular error (degrees) (c) Pumpkin 25 SCORE Forest PoseNet Dense PoseNet Nearest Neighbour CNN 0.4 0.2 30 0 0 2 4 6 8 10 12 14 16 18 Angular error (degrees) (d) Stairs Figure 7: Localization performance. These figures show our localization accuracy for both position and orientation as a cumulative histogram of errors for the entire testing set. The regression convnet outperforms the nearest neighbour feature matching which demonstrates we regress finer resolution results than given by training. Comparing to the RGB-D SCoRe Forest approach shows that our method is competitive, but outperformed by a more expensive depth approach. Our method does perform better on the hardest few frames, above the 95th percentile, with our worst error lower than the worst error from the SCoRe approach. 20 (a) Relocalization with increasing levels of motion blur. The system is able to recognize the pose as high level features such as the contour outline still exist. Blurring the landmark increases apparent contour size and the system believes it is closer. (b) Relocalization under difficult dusk and night lighting conditions. In the dusk sequences, the landmark is silhouetted against the backdrop however again the convnet seems to recognize the contours and estimate pose. (c) Relocalization with different weather conditions. PoseNet is able to effectively estimate pose in fog and rain. (d) Relocalization with significant people, vehicles and other dynamic objects. (e) Relocalization with unknown camera intrinsics: SLR with focal length 45mm (left), and iPhone 4S with focal length 35mm (right) compared to the dataset’s camera which had a focal length of 30mm. Figure 8: Robustness to challenging real life situations. Registration with point based techniques such as SIFT fails in examples (a-c), therefore ground truth measurements are not available. None of these types of challenges were seen during training. As convnets are able to understand objects and contours they are still successful at estimating pose from the building’s contour in the silhouetted examples (b) or even under extreme motion blur (a). Many of these quasi invariances were enhanced by pretraining from the scenes dataset. Test error AlexNet pretrained on Places GoogLeNet with random initialisation GoogLeNet pretrained on ImageNet GoogLeNet pretrained on Places GoogLeNet pretrained on Places, then another indoor landmark 0 20 40 60 80 100 Training epochs Figure 9: Robustness to a decreasing training baseline for the King’s College scene. Our system exhibits graceful decline in performance as fewer training samples are used. ally handles these challenges well. SfM with SIFT fails in all these cases so we were not able to generate a ground truth camera pose, however we infer the accuracy by viewing the 3D reconstruction from the predicted camera pose, and overlaying this onto the input image. 5.1. Robustness against training image spacing We demonstrate in fig. 9 that, for an outdoor scale scene, we gain little by spacing the training images more closely than 4m. The system is robust to very large spatial separation between training images, achieving reasonable performance even with only a few dozen training samples. The pose accuracy deteriorates gracefully with increased training image spacing, whereas SIFT-based SfM sharply fails after a certain threshold as it requires a small baseline [15]. 5.2. Importance of transfer learning In general convnets require large amounts of training data. We sidestep this problem by starting our pose training from a network pretrained on giant datasets such as ImageNet and Places. Similar to what has been demonstrated for classification tasks, fig. 10 shows how transfer learning can be utilised effectively between classification and complicated regression tasks. Such ‘transfer learning’ has been demonstrated elsewhere for training classifiers [18, 17, 2], but here we demonstrate transfer learning from classification to the qualitatively different task of pose regression. It is not immediately obvious that a network trained to output pose-invariant classification labels would be suitable as a starting point for a pose regressor. We find, however, that this is not a problem in practice. A possible explanation is that, in order for its output to be invariant to pose, the classifier network must keep track of pose, to better factor its effects away from identity cues. This would agree with our own findings that a network trained to output position and orientation outperforms a network trained to output only po- Figure 10: Importance of transfer learning. Shows how pretraining on large datasets gives an increase in both performance and training speed. sition. By preserving orientation information in the intermediate representations, it is better able to factor the effects of orientation out of the final position estimation. Transfer learning gives not only a large improvement in training speed, but also end performance. The relevance of data is also important. In fig. 10 the Places and ImageNet curves initially have the same performance. However, ultimately the Places pretraining performs better due to being a more relevant dataset to this localization task. 5.3. Visualising features relevant to pose Fig. 11 shows example saliency maps produced by PoseNet. The saliency map, as used in [21], is the magnitude of the gradient of the loss function with respect to the pixel intensities. This uses the sensitivity of the pose with respect to the pixels as an indicator of how important the convnet considers different parts of the image. These results show that the strongest response is observed from higher-level features such as windows and spires. However a more surprising result is that PoseNet is also very sensitive to large textureless patches such as road, grass and sky. These textureless patches may be more informative than the highest responding points because the effect of a group of pixels on the pose variable is the sum of the saliency map values over that group of pixels. This evidence points to the net being able to localize off information from these textureless surfaces, something which interest-point based features such as SIFT or SURF fail to do. The last observation is that PoseNet has an attenuated response to people and other noisy objects, effectively masking them. These objects are dynamic, and the convnet has identified them as not appropriate for localization. 5.4. Viewing the internal representation t-SNE [26] is an algorithm for embedding highdimensional data in a low dimensional space, in a way that tries to preserve Euclidean distances. It is often used, as we do here, to visualize high-dimensional feature vectors in Figure 11: Saliency maps. This figure shows the saliency map superimposed on the input image. The saliency maps suggest that the convnet exploits not only distinctive point features (à la SIFT), but also large textureless patches, which can be as informative, if not more so, to the pose. This, combined with a tendency to disregard dynamic objects such as pedestrians, enables it to perform well under challenging circumstances. (Best viewed electronically.) 120 10 8 Memory (GB) Time (seconds) 100 80 60 40 (b) (c) Figure 12: Feature vector visualisation. t-SNE visualisation of the feature vectors from a video sequence traversing an outdoor scene (King’s College) in a straight line. Colour represents time. The feature representations are generated from the convnet with weights trained on Places (a), Places then another outdoor scene, St Mary’s Church (b), Places then this outdoor scene, King’s College (c). Despite (a,b) not being trained on this scene, these visualizations suggest that it is possible to compute the pose as a simple, if non-linear, function of these representations. two dimensions. In fig. 12 we apply t-SNE to the feature vectors computed from a sequence of video frames taken by a pedestrian. As these figures show, the feature vectors are a function that smoothly varies with, and is largely oneto-one with, pose. This ‘pose manifold’ can be observed not only on networks trained on other scenes, but also networks trained on classification image sets without pose labels. This further suggests that classification convnets preserve pose information up to the final layer, regardless of whether it’s expressed in the output. However, the mapping from feature vector to pose becomes more complicated for networks not trained on pose data. Furthermore, as this manifold exists on scenes that the convnet was not trained on, the convnet must learn some generic representation of the relationship between landmarks, geometry and camera motion. This demonstrates that the feature vector that is produced from regression is able to generalize to other tasks in the same way as classification convnets. 5.5. System efficiency Fig. 13 compares system performance of PoseNet on a modern desktop computer. Our network is very scalable, as it only takes 50 MB to store the weights, and 5ms to com- 0 0 6 4 2 20 (a) SIFT Structure from Motion Nearest Neighbour CNN PoseNet 50MB 5ms 200 400 600 800 1000 1200 Number of training samples 0 0 200 400 600 800 1000 1200 Number of training samples Figure 13: Implementation efficiency. Experimental speed and memory use of the convnet regression, nearest neighbour convnet feature vector and SIFT relocalization methods. pute each pose, compared to the gigabytes and minutes for metric localization with SIFT. These values are independent of the number of training samples in the system while metric localization scales O(n2 ) with training data size [28]. For comparison matching to the convnet nearest neighbour is also shown. This requires storing feature vectors for each training frame, then perform a linear search to find the nearest neighbour for a given test frame. 6. Conclusions We present, to our knowledge, the first application of deep convolutional neural networks to end-to-end 6-DOF camera pose localization. We have demonstrated that one can sidestep the need for millions of training images by use of transfer learning from networks trained as classifiers. We showed that such networks preserve ample pose information in their feature vectors, despite being trained to produce pose-invariant outputs. Our method tolerates large baselines that cause SIFT-based localizers to fail sharply. In future work, we aim to pursue further uses of multiview geometry as a source of training data for deep pose regressors, and explore probabilistic extensions to this algorithm [12]. It is obvious that a finite neural network has an upper bound on the physical area that it can learn to localize within. We leave finding this limit to future work. References [1] S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski. Building rome in a day. Communications of the ACM, 54(10):105–112, 2011. [2] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1798–1828, 2013. [3] A. Bergamo, S. N. Sinha, and L. Torresani. Leveraging structure from motion to learn discriminative codebooks for scalable landmark classification. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 763– 770. IEEE, 2013. [4] M. Cummins and P. Newman. FAB-MAP: Probabilistic localization and mapping in the space of appearance. The International Journal of Robotics Research, 27(6):647–665, 2008. [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. IEEE, 2009. [6] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531, 2013. [7] J. Engel, T. Schöps, and D. Cremers. LSD-SLAM: Largescale direct monocular slam. In Computer Vision–ECCV 2014, pages 834–849. Springer, 2014. [8] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski. Towards internet-scale multi-view stereo. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 1434–1441. IEEE, 2010. [9] Q. Hao, R. Cai, Z. Li, L. Zhang, Y. Pang, and F. Wu. 3d visual phrases for landmark recognition. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3594–3601. IEEE, 2012. [10] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. [11] M. Kaess, H. Johannsson, R. Roberts, V. Ila, J. J. Leonard, and F. Dellaert. iSAM2: Incremental smoothing and mapping using the bayes tree. The International Journal of Robotics Research, page 0278364911430419, 2011. [12] A. Kendall and R. Cipolla. Modelling uncertainty in deep learning for camera relocalization. arXiv preprint arXiv:1509.05909, 2015. [13] G. Klein and D. Murray. Parallel tracking and mapping for small ar workspaces. In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on, pages 225–234. IEEE, 2007. [14] Y. Li, N. Snavely, D. Huttenlocher, and P. Fua. Worldwide pose estimation using 3d point clouds. In Computer Vision– ECCV 2012, pages 15–29. Springer, 2012. [15] D. G. Lowe. Distinctive image features from scaleinvariant keypoints. International journal of computer vision, 60(2):91–110, 2004. [16] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison. DTAM: Dense tracking and mapping in real-time. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2320–2327. IEEE, 2011. [17] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1717–1724. IEEE, 2014. [18] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on, pages 512–519. IEEE, 2014. [19] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013. [20] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon. Scene coordinate regression forests for camera relocalization in RGB-D images. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 2930–2937. IEEE, 2013. [21] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013. [22] N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: exploring photo collections in 3d. In ACM transactions on graphics (TOG), volume 25, pages 835–846. ACM, 2006. [23] N. Sünderhauf, F. Dayoub, S. Shirazi, B. Upcroft, and M. Milford. On the performance of convnet features for place recognition. arXiv preprint arXiv:1501.04158, 2015. [24] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. [25] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1653–1660. IEEE, 2014. [26] L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(25792605):85, 2008. [27] J. Wang, H. Zha, and R. Cipolla. Coarse-to-fine vision-based localization by indexing scale-invariant features. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 36(2):413–422, 2006. [28] C. Wu. Towards linear-time incremental structure from motion. In 3D Vision-3DV 2013, 2013 International Conference on, pages 127–134. IEEE, 2013. [29] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Advances in Neural Information Processing Systems, pages 487–495, 2014. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu ∗,1 Ryan Kiros*,1 Richard Zemel1 Ruslan Salakhutdinov1 Raquel Urtasun1 Antonio Torralba2 Sanja Fidler1 1 University of Toronto 2 Massachusetts Institute of Technology arXiv:1506.06724v1 [cs.CV] 22 Jun 2015 {yukun,rkiros,zemel,rsalakhu,urtasun,fidler}@cs.toronto.edu, torralba@csail.mit.edu Abstract Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we exploit a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for. Figure 1: Shot from the movie Gone Girl, along with the subtitle, aligned with the book. We reason about the visual and dialog (text) alignment between the movie and a book. Books provide us with very rich, descriptive text that conveys both fine-grained visual details (how people or scenes look like) as well as high-level semantics (what people think and feel, and how their states evolve through a story). This source of knowledge, however, does not come with associated visual information that would enable us to ground it with descriptions. Grounding descriptions in books to vision would allow us to get textual explanations or stories behind visual information rather than simplistic captions available in current datasets. It can also provide us with extremely large amount of data (with tens of thousands books available online). In this paper, we exploit the fact that many books have been turned into movies. Books and their movie releases have a lot of common knowledge as well as they are complementary in many ways. For instance, books provide detailed descriptions about the intentions and mental states of the characters, while movies are better at capturing visual aspects of the settings. The first challenge we need to address, and the focus of this paper, is to align books with their movie releases in order to obtain rich descriptions for the visual content. We aim to align the two sources with two types of information: visual, where the goal is to link a movie shot to a book paragraph, and dialog, where we want to find correspondences between sentences in the movie’s subtitle and sentences in the book. We formulate the problem of movie/book alignment as finding correspondences between shots in the movie as well as dialog sentences in the subtitles and sentences in the book (Fig. 1). We introduce a novel sentence similarity measure based on a neural sen- 1. Introduction A truly intelligent machine needs to not only parse the surrounding 3D environment, but also understand why people take certain actions, what they will do next, what they could possibly be thinking, and even try to empathize with them. In this quest, language will play a crucial role in grounding visual information to high-level semantic concepts. Only a few words in a sentence may convey really rich semantic information. Language also represents a natural means of interaction between a naive user and our vision algorithms, which is particularly important for applications such as social robotics or assistive driving. Combining images or videos with language has gotten significant attention in the past year, partly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗ Denotes equal contribution 1 tence embedding trained on millions of sentences from a large corpus of books. On the visual side, we extend the neural image-sentence embeddings to the video domain and train the model on DVS descriptions of movie clips. Our approach combines different similarity measures and takes into account contextual information contained in the nearby shots and book sentences. Our final alignment model is formulated as an energy minimization problem that encourages the alignment to follow a similar timeline. To evaluate the book-movie alignment model we collected a dataset with 11 movie/book pairs annotated with 2,070 shot-to-sentence correspondences. We demonstrate good quantitative performance and show several qualitative examples that showcase the diversity of tasks our model can be used for. The alignment model can have multiple applications. Imagine an app which allows the user to browse the book as the scenes unroll in the movie: perhaps its ending or acting are ambiguous, and one would like to query the book for answers. Vice-versa, while reading the book one might want to switch from text to video, particularly for the juicy scenes. We also show other applications of learning from movies and books such as book retrieval (finding the book that goes with a movie and finding other similar books), and captioning CoCo images with story-like descriptions. 2. Related Work Most effort in the domain of vision and language has been devoted to the problem of image captioning. Older work made use of fixed visual representations and translated them into textual descriptions [6, 16]. Recently, several approaches based on RNNs emerged, generating captions via a learned joint image-text embedding [13, 11, 36, 21]. These approaches have also been extended to generate descriptions of short video clips [35]. In [24], the authors go beyond describing what is happening in an image and provide explanations about why something is happening. For text-to-image alignment, [15, 7] find correspondences between nouns and pronouns in a caption and visual objects using several visual and textual potentials. Lin et al. [17] does so for videos. In [11], the authors use RNN embeddings to find the correspondences. [37] combines neural embeddings with soft attention in order to align the words to image regions. Early work on movie-to-text alignment include dynamic time warping for aligning movies to scripts with the help of subtitles [5, 4]. Sankar et al. [28] further developed a system which identified sets of visual and audio features to align movies and scripts without making use of the subtitles. Such alignment has been exploited to provide weak labels for person naming tasks [5, 30, 25]. Closest to our work is [34], which aligns plot synopses to shots in the TV series for story-based content retrieval. This work adopts a similarity function between sentences in plot synopses and shots based on person identities and keywords in subtitles. Our work differs with theirs in several important aspects. First, we tackle a more challenging problem of movie/book alignment. Unlike plot synopsis, which closely follow the storyline of movies, books are more verbose and might vary in the storyline from their movie release. Furthermore, we use learned neural embeddings to compute the similarities rather than hand-designed similarity functions. Parallel to our work, [33] aims to align scenes in movies to chapters in the book. However, their approach operates on a very coarse level (chapters), while ours does so on the sentence/paragraph level. Their dataset thus evaluates on 90 scene-chapter correspondences, while our dataset draws 2,070 shot-to-sentences alignments. Furthermore, the approaches are inherently different. [33] matches the presence of characters in a scene to those in a chapter, as well as uses hand-crafted similarity measures between sentences in the subtitles and dialogs in the books, similarly to [34]. Rohrbach et al. [27] recently released the Movie Description dataset which contains clips from movies, each time-stamped with a sentence from DVS (Descriptive Video Service). The dataset contains clips from over a 100 movies, and provides a great resource for the captioning techniques. Our effort here is to align movies with books in order to obtain longer, richer and more high-level video descriptions. We start by describing our new dataset, and then explain our proposed approach. 3. The MovieBook and BookCorpus Datasets We collected two large datasets, one for movie/book alignment and one with a large number of books. The MovieBook Dataset. Since no prior work or data exist on the problem of movie/book alignment, we collected a new dataset with 11 movies along with the books on which they were based on. For each movie we also have a subtitle file, which we parse into a set of time-stamped sentences. Note that no speaker information is provided in the subtitles. We automatically parse each book into sentences, paragraphs (based on indentation in the book), and chapters (we assume a chapter title has indentation, starts on a new page, and does not end with an end symbol). Our annotators had the movie and a book opened side by side. They were asked to iterate between browsing the book and watching a few shots/scenes of the movie, and trying to find correspondences between them. In particular, they marked the exact time (in seconds) of correspondence in the movie and the matching line number in the book file, indicating the beginning of the matched sentence. On the video side, we assume that the match spans across a shot (a video unit with smooth camera motion). If the match was longer in duration, the annotator also indicated the ending time of the match. Similarly for the book, if more sentences BOOK MOVIE 148,340 48,946 69,824 78,596 40,140 133,241 143,631 112,978 135,529 10,640 58,793 # unique words 3,849 1,833 1,704 2,363 1,360 3,043 4,632 2,949 3,685 470 1,580 avg. # words per sent. 15 14 10 15 18 17 16 19 11 20 10 max # words per sent. 153 90 68 227 115 119 422 192 85 173 74 # paragraphs 3,927 2,082 3,189 2,925 637 2,760 3,945 2,236 5,223 167 2,345 980,658 9,032 15 156 29,436 Title # sent. # words Gone Girl Fight Club No Country for Old Men Harry Potter and the Sorcerers Stone Shawshank Redemption The Green Mile American Psycho One Flew Over the Cuckoo Nest The Firm Brokeback Mountain The Road 12,603 4,229 8,050 6,458 2,562 9,467 11,992 7,103 15,498 638 6,638 All 85,238 ANNOTATION 2,604 2,365 1,348 2,647 1,252 2,350 1,012 1,671 2,423 1,205 1,108 # sent. in subtitles 2,555 1,864 889 1,227 1,879 1,846 1,311 1,553 1,775 1,228 782 # dialog align. 76 104 223 164 44 208 278 64 82 80 126 # visual align. 106 42 47 73 12 102 85 25 60 20 49 19,985 16,909 1,449 621 # shots Table 1: Statistics for our MovieBook Dataset with ground-truth for alignment between books and their movie releases. # of books 11,038 # of sentences 74,004,228 # of words 984,846,357 # of unique words 1,316,420 mean # of words per sentence 13 median # of words per sentence 11 Table 2: Summary statistics of our BookCorpus dataset. We use this corpus to train the sentence embedding model. matched, the annotator indicated from which to which line a match occurred. Each alignment was also tagged, indicating whether it was a visual, dialogue, or an audio match. Note that even for dialogs, the movie and book versions are semantically similar but not exactly the same. Thus deciding on what defines a match or not is also somewhat subjective and may slightly vary across our annotators. Altogether, the annotators spent 90 hours labeling 11 movie/book pairs, locating 2,070 correspondences. Table 1 presents our dataset, while Fig. 8 shows a few ground-truth alignments. One can see the complexity and diversity of the data: the number of sentences per book vary from 638 to 15,498, even though the movies are similar in duration. This indicates a huge diversity in descriptiveness across literature, and presents a challenge for matching. The sentences also vary in length, with the sentences in Brokeback Mountain being twice as long as those in The Road. The longest sentence in American Psycho has 422 words and spans over a page in the book. Aligning movies with books is challenging even for humans, mostly due to the scale of the data. Each movie is on average 2h long and has 1,800 shots, while a book has on average 7,750 sentences. Books also have different styles of writing, formatting, different and challenging language, slang (going vs goin’, or even was vs ’us), etc. As one can see from Table 1, finding visual matches turned out to be particularly challenging. This is because the visual descriptions in books can be either very short and hidden within longer paragraphs or even within a longer sentence, or very verbose – in which case they get obscured with the surrounding text – and are hard to spot. Of course, how close the movie follows the book is also up to the director, which can be seen through the number of alignments that our annotators found across different movie/books. lished authors. We only included books that had more than 20K words in order to filter out perhaps noisier shorter stories. The dataset has books in 16 different genres, e.g., Romance (2,865 books), Fantasy (1,479), Science fiction (786), Teen (430), etc. Table 2 highlights the summary statistics of our book corpus. The BookCorpus Dataset. In order to train our sentence similarity model we collected a corpus of 11,038 books from the web. These are free books written by yet unpub- In order to score the similarity between two sentences, we exploit our architecture for learning unsupervised representations of text [14]. The model is loosely inspired by 4. Aligning Books and Movies Our approach aims to align a movie with a book by exploiting visual information as well as dialogs. We take shots as video units and sentences from subtitles to represent dialogs. Our goal is to match these to the sentences in the book. We propose several measures to compute similarities between pairs of sentences as well as shots and sentences. We use our novel deep neural embedding trained on our large corpus of books to predict similarities between sentences. Note that an extended version of the sentence embedding is described in detail in [14] showing how to deal with million-word vocabularies, and demonstrating its performance on a large variety of NLP benchmarks. For comparing shots with sentences we extend the neural embedding of images and text [13] to operate in the video domain. We next develop a novel contextual alignment model that combines information from various similarity measures and a larger time-scale in order to make better local alignment predictions. Finally, we propose a simple pairwise Conditional Random Field (CRF) that smooths the alignments by encouraging them to follow a linear timeline, both in the video and book domain. We first explain our sentence, followed by our joint video to text embedding. We next propose our contextual model that combines similarities and discuss CRF in more detail. 4.1. Skip-Thought Vectors Figure 2: Sentence neural embedding [14]. Given a tuple (si−1 , si , si+1 ) of consecutive sentences in text, where si is the i-th sentence, we encode si and aim to reconstruct the previous si−1 and the following sentence si+1 . Unattached arrows are connected to the encoder output. Colors depict which components share parameters. heosi is the end of sentence token. he drove down the street off into the distance . the most effective way to end the battle . he started the car , left the parking lot and merged onto the highway a few miles down the road . he shut the door and watched the taxi drive off . she watched the lights flicker through the trees as the men drove toward the road . he jogged down the stairs , through the small lobby , through the door and into the street . a messy business to be sure , but necessary to achieve a fine and noble end . they saw their only goal as survival and logically planned a strategy to achieve it . there would be far fewer casualties and far less destruction . the outcome was the lisbon treaty . Table 3: Qualitative results from the sentence embedding model. For each query sentence on the left, we retrieve the 4 nearest neighbor sentences (by inner product) chosen from books the model has not seen before. the skip-gram [22] architecture for learning representations of words. In the word skip-gram model, a word wi is chosen and must predict its surrounding context (e.g. wi+1 and wi−1 for a context window of size 1). Our model works in a similar way but at the sentence level. That is, given a sentence tuple (si−1 , si , si+1 ) our model first encodes the sentence si into a fixed vector, then conditioned on this vector tries to reconstruct the sentences si−1 and si+1 , as shown in Fig. 2. The motivation for this architecture is inspired by the distributional hypothesis: sentences that have similar surrounding context are likely to be both semantically and syntactically similar. Thus, two sentences that have similar syntax and semantics are likely to be encoded to a similar vector. Once the model is trained, we can map any sentence through the encoder to obtain vector representations, then score their similarity through an inner product. The learning signal of the model depends on having contiguous text, where sentences follow one another in sequence. A natural corpus for training our model is thus a large collection of books. Given the size and diversity of genres, our BookCorpus allows us to learn very general representations of text. For instance, Table 3 illustrates the nearest neighbours of query sentences, taken from held out books that the model was not trained on. These qualitative results demonstrate that our intuition is correct, with resulting nearest neighbors corresponds largely to syntactically and semantically similar sentences. Note that the sentence embedding is general and can be applied to other domains not considered in this paper, which is explored in [14]. To construct an encoder, we use a recurrent neural network, inspired by the success of encoder-decoder models for neural machine translation [10, 2, 1, 31]. Two kinds of activation functions have recently gained traction: long short-term memory (LSTM) [9] and the gated recurrent unit (GRU) [3]. Both types of activation successfully solve the vanishing gradient problem, through the use of gates to control the flow of information. The LSTM unit explicity employs a cell that acts as a carousel with an identity weight. The flow of information through a cell is controlled by input, output and forget gates which control what goes into a cell, what leaves a cell and whether to reset the contents of the cell. The GRU does not use a cell but employs two gates: an update and a reset gate. In a GRU, the hidden state is a linear combination of the previous hidden state and the proposed hidden state, where the combination weights are controlled by the update gate. GRUs have been shown to perform just as well as LSTM on several sequence prediction tasks [3] while being simpler. Thus, we use GRU as the activation function for our encoder and decoder RNNs. Suppose that we are given a sentence tuple (si−1 , si , si+1 ), and let wit denote the t-th word for si and let xti be its word embedding. We break the model description into three parts: the encoder, decoder and objective function. Encoder. Let wi1 , . . . , wiN denote words in sentence si with N the number of words in the sentence. The encoder produces a hidden state hti at each time step which forms the representation of the sequence wi1 , . . . , wit . Thus, the hidden state hN i is the representation of the whole sentence. The GRU produces the next hidden state as a linear combination of the previous hidden state and the proposed state update (we drop subscript i): ht = (1 − zt ) ht−1 + zt h̄t (1) where h̄t is the proposed state update at time t, zt is the update gate and ( ) denotes a component-wise product. The update gate takes values between zero and one. In the extreme cases, if the update gate is the vector of ones, the previous hidden state is completely forgotten and ht = h̄t . Alternatively, if the update gate is the zero vector, than the hidden state from the previous time step is simply copied over, that is ht = ht−1 . The update gate is computed as zt = σ(Wz xt + Uz ht−1 ) (2) where Wz and Uz are the update gate parameters. The proposed state update is given by h̄t = tanh(Wxt + U(rt ht−1 )) (3) where rt is the reset gate, which is computed as rt = σ(Wr xt + Ur ht−1 ) (4) If the reset gate is the zero vector, than the proposed state update is computed only as a function of the current word. Thus after iterating this equation sequence for each word, we obtain a sentence vector hN i = hi for sentence si . Decoder. The decoder computation is analogous to the encoder, except that the computation is conditioned on the sentence vector hi . Two separate decoders are used, one for the previous sentence si−1 and one for the next sentence si+1 . These decoders use different parameters to compute their hidden states but both share the same vocabulary matrix V that takes a hidden state and computes a distribution over words. Thus, the decoders are analogous to an RNN language model but conditioned on the encoder sequence. Alternatively, in the context of image caption generation, the encoded sentence hi plays a similar role as the image. We describe the decoder for the next sentence si+1 (computation for si−1 is identical). Let hti+1 denote the hidden state of the decoder at time t. The update and reset gates for the decoder are given as follows (we drop i + 1): zt t r = σ(Wzd xt−1 + Udz ht−1 + Cz hi ) (5) = σ(Wrd xt−1 (6) the hidden state t hti+1 + Cr hi ) d t−1 t = (1 − z ) d t + U (r t−1 h +z t it = σ(Wxi xt + Whi mt−1 + Wci ct−1 ) (11) t = σ(Wxf xt + Whf mt−1 + Wcf ct−1 ) (12) f at c t−1 h h̄ (8) t <t t P (wi+1 |wi+1 , hi ) ∝ exp(vwi+1 hti+1 ) (9) t where vwi+1 denotes the row of V corresponding to the t word of wi+1 . An analogous computation is performed for the previous sentence si−1 . Objective. Given (si−1 , si , si+1 ), the objective optimized is the sum of log-probabilities for the next and previous sentences conditioned on the representation of the encoder: X X t <t t <t logP (wi+1 |wi+1 , hi ) + logP (wi−1 |wi−1 , hi ) t (10) The total objective is the above summed over all such training tuples. Adam algorithm [12] is used for optimization. = tanh(Wxc xt + Whc mt−1 ) t = f t t c t−1 +i t t o ) + Chi ) (7) t t Given hti+1 , the probability of word wi+1 given the previous t − 1 words and the encoder vector is t The model above describes how to obtain a similarity score between two sentences, whose representations are learned from millions of sentences in books. We now discuss how to obtain similarities between shots and sentences. Our approach closely follows the image-sentence ranking model proposed by [13]. In their model, an LSTM is used for encoding a sentence into a fixed vector. A linear mapping is applied to image features from a convolutional network. A score is computed based on the inner product between the normalized sentence and image vectors. Correct image-sentence pairs are trained to have high score, while incorrect pairs are assigned low scores. In our case, we learn a visual-semantic embedding between movie clips and their DVS description. DVS (“Descriptive Video Service”) is a service that inserts audio descriptions of the movie between the dialogs in order to enable the visually impaired to follow the movie like anyone else. We used the movie description dataset of [27] for learning our embedding. This dataset has 94 movies, and 54,000 described clips. We represent each movie clip as a vector corresponding to mean-pooled features across each frame in the clip. We used the GoogLeNet architecture [32] as well as hybrid-CNN [38] for extracting frame features. For DVS, we pre-processed the descriptions by removing names and replacing these with a someone token. The LSTM architecture in this work is implemented using the following equations. As before, we represent a word embedding at time t of a sentence as xt : is then computed as: h̄ = tanh(W x hti+1 + Udr ht−1 4.2. Visual-semantic embeddings of clips and DVS a (14) t−1 = σ(Wxo x + Who m mt = ot (13) t t + Wco c ) tanh(ct ) (15) (16) where (σ) denotes the sigmoid activation function and ( ) indicates component-wise multiplication. The states (it , f t , ct , ot , mt ) correspond to the input, forget, cell, output and memory vectors, respectively. If the sentence is of length N , then the vector mN = m is the vector representation of the sentence. Let q denote a movie clip vector, and let v = WI q be the embedding of the movie clip. We define a scoring function s(m, v) = m · v, where m and v are first scaled to have unit norm (making s equivalent to cosine similarity). We then optimize the following pairwise ranking loss: XX min max{0, α − s(m, v) + s(m, vk )} (17) θ + m k XX v k max{0, α − s(v, m) + s(v, mk )}, with mk a contrastive (non-descriptive) sentence vector for a clip embedding v, and vice-versa with vk . We train our model with stochastic gradient descent without momentum. 4.3. Context aware similarity We employ the clip-sentence embedding to compute similarities between each shot in the movie and each sentence in the book. For dialogs, we use several similarity measures each capturing a different level of semantic similarity. We compute BLEU [23] between each subtitle and book sentence to identify nearly identical matches. Similarly to [34], we use a tf-idf measure to find near duplicates but weighing down the influence of the less frequent words. Finally, we use our sentence embedding learned from books to score pairs of sentences that are semantically similar but may have a very different wording (i.e., paraphrasing). These similarity measures indicate the alignment between the two modalities. However, at the local, sentence level, alignment can be rather ambiguous. For example, despite being a rather dark book, Gone Girl contains 15 occurrences of the sentence “I love you”. We exploit the fact that a match is not completely isolated but that the sentences (or shots) around it are also to some extent similar. We design a context aware similarity measure that takes into account all individual similarity measures as well as a fixed context window in both, the movie and book domain, and predicts a new similarity score. We stack a set of M similarity measures into a tensor S(i, j, m), where i, j, and m are the indices of sentences in the subtitle, in the book, and individual similarity measures, respectively. In particular, we use M = 9 similarities: visual and sentence embedding, BLEU1-5, tf-idf, and a uniform prior. We want to predict a combined score score(i, j) = f (S(I, J, M)) at each location (i, j) based on all measurements in a fixed volume defined by I around i, J around j, and 1, . . . , M . Evaluating the function f (·) at each location (i, j) on a 3-D tensor S is very similar to applying a convolution using a kernel of appropriate size. This motivates us to formulate the function f (·) as a deep convolutional neural network (CNN). In this paper, we adopt a 3-layer CNN as illustrated in Figure 3. We adopt the ReLU non-linearity with dropout to regularize our model. We optimize the cross-entropy loss over the training set using Adam algorithm. 4.4. Global Movie/Book Alignment So far, each shot/sentence was matched independently. However, most shots in movies and passages in the books follow a similar timeline. We would like to incorporate this prior into our alignment. In [34], the authors use dynamic time warping by enforcing that the shots in the movie can only match forward in time (to plot synopses in their case). However, the storyline of the movie and book can have crossings in time (Fig. 8), and the alignment might contain Figure 3: Our CNN for context-aware similarity computation. It has 3 conv. layers and a sigmoid layer on top. giant leaps forwards or backwards. Therefore, we formulate a movie/book alignment problem as inference in a Conditional Random Field that encourages nearby shots/dialog alignments to be consistent. Each node yi in our CRF represents an alignment of the shot in the movie with its corresponding subtitle sentence to a sentence in the book. Its state space is thus the set of all sentences in the book. The CRF energy of a configuration y is formulated as: − log p(x,y; ω) = K X ωu φu (yi ) + i=1 K X X ωp ψp (yi , yj ) i=1 j∈N (i) where K is the number of nodes (shots), and N (i) the left and right neighbor of yi . Here, φu (·) and ψp (·) are unary and pairwise potentials, respectively, and ω = (ωu , ωp ). We directly use the output of the CNN from 4.3 as the unary potential φu (·). For the pairwise potential, we measure the time span ds (yi , yj ) between two neighbouring sentences in the subtitle and the distance db (yi , yj ) of their state space in the book. One pairwise potential is defined as: ψp (yi , yj ) = (ds (yi , yj ) − db (yi , yj )) 2 2 (ds (yi , yj ) − db (yi , yj )) + σ 2 (18) Here σ 2 is a robustness parameter to avoid punishing giant leaps too harsh. Both ds and db are normalized to [0, 1]. In addition, we also employ another pairwise poten(d (y ,y ))2 tial ψq (yi , yj ) = (d (yb ,yi ))j2 +σ2 to encourage state consisi j b tency between nearby nodes. This potential is helpful when there is a long silence (no dialog) in the movie. Inference. Our CRF is a chain, thus exact inference is possible using dynamic programming. We also prune some states that are very far from the uniform alignment (over 1/3 length of the book) to further speed up computation. Learning. Since ground-truth is only available for a sparse set of shots, we regard the states of unobserved nodes as hidden variables and learn the CRF weights with [29]. 5. Experimental Evaluation We evaluate our model on our dataset of 11 movie/book pairs. We train the parameters in our model (CNN and CRF) Running Times. We show the typical running time of each component in our model in Table 5. For each moviebook pair, calculating BLEU score takes most of the time. Note that BLEU does not contribute significantly to the performance and is of optional use. With respect to the rest, extracting visual features VIS (mean pooling GoogleNet features over the shot frames) and SCENE features (mean pooling hybrid-CNN features [38] over the shot frames), 49.5 Shawshank ... American Psy. The Road The Firm The Firm Shawshank ... Green Mile Harry Potter 68.5 Shawshank ... The Firm 50.6 Green Mile Shawshank ... The Firm 58.0 50.7 54.1 46.4 No Country ... Brokeback ... No Country ... Brokeback ... American Psy. Brokeback ... 59.1 53.9 53.4 Harry Potter No Country ... One Flew... Shawshank ... Shawshank ... The Road One Flew... 54.8 51.3 76.9 71.4 50.9 Fight Club 51.9 38.7 77.8 71.5 Green Mile 73.7 46.8 52.6 Shawshank ... 55.9 Brokeback ... 49.7 The Firm The Road Shawshank ... The Road Fight Club One Flew... 52.2 56.0 The Road 100.0 100.0 73.9 54.8 The Firm 100.0 The Road Fight Club Shawshank ... One Flew... 75.0 60.9 36.7 39.0 79.0 61.4 42.7 38.0 53.1 79.1 62.0 43.0 39.1 53.5 80.8 43.6 38.9 39.5 Fight Club One Flew... The Firm The Firm Shawshank ... Brokeback ... 39.6 54.9 66.0 100.0 Brokeback ... The Firm No Country... 84.0 100.0 The Firm Shawshank ... 40.1 39.7 55.5 100.0 45.1 One Flew... 100.0 One Flew... The Road 40.5 45.2 No Country ... American Psy. Brokeback ... One Flew... Brokeback ... 100.0 One Flew... American Psy. 42.5 The Firm Harry Potter No Country ... The Road Evaluating the performance of movie/book alignment is an interesting problem on its own. This is because our ground-truth is far from exhaustive – around 200 correspondences were typically found between a movie and its book, and likely a number of them got missed. Thus, evaluating the precision is rather tricky. We thus focus our evaluation on recall, similar to existing work on retrieval. For each shot that has a GT correspondence in book, we check whether our prediction is close to the annotated one. We evaluate recall at the paragraph level, i.e., we say that the GT paragraph was recalled, if our match was at most 3 paragraphs away, and the shot was at most 5 subtitle sentences away. As a noisier measure, we also compute recall and precision at multiple alignment thresholds and report AP (avg. prec.). The results are presented in Table 4. Columns show different instantiations of our model: we show the leave-onefeature-out setting (∅ indicates that all features were used), compare how different depths of the context-aware CNN influence the performance, and compare it to our full model (CRF) in the last column. We get the highest boost by adding more layers to the CNN – recall improves by 14%, and AP doubles. Generally, each feature helps performance. Our sentence embedding (BOOK) helps by 4%, while noisier video-text embedding helps by 2% in recall. CRF which encourages temporal smoothness generally helps (but not for all movies), bringing additional 2%. We also show how a uniform timeline performs on its own. That is, for each shot (measured in seconds) in the movie, we find the sentence at the same location (measured in lines) in the book. We add another baseline to evaluate the role of context in our model. Instead of using our CNN that considers contextual information, we build a linear SVM to combine different similarity measures in a single node (shot) – the final similarity is used as a unary potential in our CRF alignment model. The Table shows that our CNN contextual model outperforms the SVM baseline by 30% in recall, and doubles the AP. We plot alignment for a few movies in Fig. 8. 100.0 American Psy.o Harry Potter Green Mile 5.1. Movie/Book Alignment 45.4 One Flew... 100.0 No Country ... Harry Potter Fight Club Fight Club American Psy. American Psy. One Flew... BOOKS No Country ... MOVIE Green Mile on Gone Girl, and test our performance on the remaining 10 movies. In terms of training speed, our video-text model “watches” 1,440 movies per day and our sentence model reads 870 books per day. We also show various qualitative results demonstrating the power of our approach. We provide more results in the Appendix of the paper. 45.8 45.8 Table 6: Book “retrieval”. For a movie (left), we rank books wrt to their alignment similarity with the movie. We normalize similarity to be 100 for the highest scoring book. takes most of the time (about 80% of the total time). We also report training times for our contextual model (CNN) and the CRF alignment model. Note that the times are reported for one movie/book pair since we used only one such pair to train all our CNN and CRF parameters. We chose Gone Girl for training since it had the best balance between the dialog and visual correspondences. 5.2. Describing Movies via the Book We next show qualitative results of our alignment. In particular, we run our model on each movie/book pair, and visualize the passage in the book that a particular shot in the movie aligns to. We show best matching paragraphs as well as a paragraph before and after. The results are shown in Fig. 8. One can see that our model is able to retrieve a semantically meaningful match despite large dialog deviations from those in the book, and the challenge of matching a visual representation to the verbose text in the book. Figure 4: Describing movie clips via the book: we align the movie to the book, and show a shot from the movie and its corresponding paragraph (plus one before and after) from the book. Figure 5: We can use our model to caption movies via a corpus of books. Top: A shot from American Pyscho is captioned with paragraphs from the Fight Club, and a shot from Harry Potter with paragraphs from Fight Club. Middle and Bottom: We match shots from Avatar and Batman Begins against 300 books from our BookCorpus, and show the best matched paragraph. Fight Club The Green Mile Harry Potter and the Sorcerers Stone American Psycho One Flew Over the Cuckoo Nest Shawshank Redemption The Firm Brokeback Mountain The Road No Country for Old Men AP Recall AP Recall AP Recall AP Recall AP Recall AP Recall AP Recall AP Recall AP Recall AP Recall Mean Recall AP UNI SVM 1.22 2.36 0.00 0.00 0.00 0.00 0.00 0.27 0.00 1.01 0.00 1.79 0.05 1.38 2.36 27.0 0.00 1.12 0.00 1.12 3.88 0.40 0.73 10.38 14.05 51.42 10.30 44.35 14.78 34.25 5.68 25.25 8.94 46.43 4.46 18.62 24.91 74.00 13.77 41.90 12.11 33.46 ∅ 0.45 12.26 14.12 62.46 8.09 51.05 16.76 67.12 8.14 41.41 8.60 78.57 7.91 33.79 16.55 88.00 6.58 43.02 9.00 48.90 BLEU 0.41 12.74 14.09 60.57 8.18 52.30 17.22 66.58 6.27 34.34 8.89 76.79 8.66 36.55 17.82 92.00 7.83 48.04 9.39 49.63 38.01 10.97 52.66 9.62 52.95 9.88 1 layer CNN w/o one feature TF-IDF BOOK VIS 0.40 0.50 0.64 11.79 11.79 12.74 6.92 10.12 9.83 53.94 57.10 55.52 5.66 7.84 7.95 46.03 48.54 48.54 12.29 14.88 14.95 60.82 64.66 63.56 1.93 8.49 8.51 32.32 36.36 37.37 4.35 7.99 8.91 73.21 73.21 78.57 2.02 6.22 7.15 26.90 23.45 26.90 14.60 15.16 15.58 86.00 86.00 88.00 3.04 5.11 5.47 32.96 38.55 37.99 8.22 9.40 9.35 46.69 47.79 51.10 47.07 5.94 48.75 8.57 50.03 8.83 SCENE 0.50 11.79 13.00 60.57 8.04 49.37 15.68 66.58 9.32 36.36 9.22 75.00 7.25 30.34 15.41 86.00 6.09 42.46 8.63 49.26 PRIOR 0.48 11.79 14.42 62.78 8.20 52.72 16.54 67.67 9.04 40.40 7.86 78.57 7.26 31.03 16.21 87.00 7.00 44.13 9.40 48.53 50.77 9.31 52.46 9.64 CNN-3 CRF 1.95 17.92 28.80 74.13 27.17 76.57 34.32 81.92 14.83 49.49 19.33 94.64 18.34 37.93 31.80 98.00 19.80 65.36 28.75 71.69 5.17 19.81 27.60 78.23 23.65 78.66 32.87 80.27 21.13 54.55 19.96 96.79 20.74 44.83 30.58 100.00 19.58 65.10 30.45 72.79 66.77 22.51 69.10 23.17 Table 4: Performance of our model for the movies in our dataset under different settings and metrics. Per movie-book pair BLEU 6h TF 10 min BOOK 3 min VIS 2h SCENE 1h CNN (training) 3 min CNN (inference) 0.2 min CRF (training) 5h CRF (inference) 5 min Table 5: Running time for our model per one movie/book pair. 5.3. Book “Retrieval” 6. Conclusion In this experiment, we compute alignment between a movie and all (test) 10 books, and check whether our model retrieves the correct book. Results are shown in Table 6. Under each book we show the computed similarity. In particular, we use the energy from the CRF, and scale all similarities relative to the highest one (100). Notice that our model retrieves the correct book for each movie. Describing a movie via other books. We can also caption movies by matching shots to paragraphs in a corpus of books. Here we do not encourage a linear timeline (CRF) since the stories are unrelated, and we only match at the local, shot-paragraph level. We show a description for American Psycho borrowed from the book Fight Club in Fig. 5. In this paper, we explored a new problem of aligning a book to its movie release. We proposed an approach that computes several similarities between shots and dialogs and the sentences in the book. We exploited our new sentence embedding in order to compute similarities between sentences. We further extended the image-text neural embeddings to video, and proposed a context-aware alignment model that takes into account all the available similarity information. We showed results on a new dataset of movie/book alignments as well as several quantitative results that showcase the power and potential of our approach. 5.4. The CoCoBook: Writing Stories for CoCo Our next experiment shows that our model is able to “generate” descriptive stories for (static) images. In particular we used the image-text embedding from [13] and generated a simple caption for an image. We used this caption as a query, and used our sentence embedding trained on books to find top 10 nearest sentences (sampled from a few hundred thousand from BookCorpus). We re-ranked these based on the 1-gram precision of non-stop words. Given the best result, we return the sentence as well as the 2 sentences before and after it in the book. The results are in Fig. 6. Our sentence embedding is able to retrieve semantically meaningful stories to explain the images. Acknowledgments We acknowledge the support from NSERC, CIFAR, Samsung, Google, and ONR-N00014-14-1-0232. We also thank Lea Jensterle for helping us with elaborate annotation, and Relu Patrascu for his help with numerous infrastructure related problems. Appendix In the Appendix we provide more qualitative results. A. Qualitative Movie-Book Alignment Results We show a few qualitative examples of alignment in Fig. 8. In this experiment, we show results obtained with our full model (CRF). For a chosen shot (a node in the CRF) we show the corresponding paragraph in the book. the club was a little emptier than i would have expected for the late afternoon , and the bartender , in red waistcoat and bowtie , was busy wiping down his counter , replacing peanuts and putting out new coasters . a television with the latest la liga news was hung in an upper corner , and behind him , rows of bottles were reflected in a giant bar mirror . above the stools , a pergola-type overhead structure held rows of wine glasses . it was a classy place , with ferns in the corner , and not the kind of bar to which i was accustomed . my places usually had a more ... relaxed feel . he felt like an idiot for yelling at the child , but his frustration and trepidation was getting the better of him . he glanced toward the shadowed hall and quickly nodded toward melissa before making his way forward . he came across more children sitting upon a couch in the living room . they watched him , but did n’t move and did n’t speak . his skin started to feel like hundreds of tiny spiders were running up and down it and he hurried on . a few miles before tioga road reached highway 395 and the town of lee vining , smith turned onto a narrow blacktop road . on either side were parched , grassy open slopes with barbed-wire fences marking property lines . cattle and horses grazed under trees whose black silhouettes stood stark against the gold-velvet mountains . marty burst into song : “ home , home on the range , where the deer and the antelope play ! where seldom is heard a discouraging word and the skies are not cloudy all day ! ” “number seventy-three , second to last from the corner . ’ adam slowed the porsche as he approached the quaint-he could think of no other word to use , even though “quaint” was one he normally , manfully , avoided-townhouse , coming to a halt beside a sleek jaguar sedan . it was a quiet street , devoid of traffic at this hour on a monday night . in the bluish-tinted light of a corner street lamp , he developed a quick visual impression of wrought-iron railings on tidy front stoops , window boxes full of bright chrysanthemums , beveled glass in bay windows , and lace curtains . townhouses around here didn’t rent cheaply , he could n’t help but observe . Figure 6: CoCoBook: We generate a caption for a CoCo image via [13] and retrieve its best matched sentence (+ 2 before and after) from a large book corpus. One can see a semantic relevance of the retrieved passage to the image. Figure 7: Alignment results of our model (bottom) compared to ground-truth alignment (top). In ground-truth, blue lines indicate visual matches, and magenta are the dialog matches. Yellow lines indicate predicted alignments. We can see that some dialogs in the movies closely follow the book and thus help with the alignment. This is particularly important since the visual information is not as strong. Since the text around the dialogs typically describe the scene, the dialogs thus help us ground the visual information contained in the description and the video. B. Borrowing “Lines” from Other Books We show a few qualitative examples of top-scoring matches for shot in a movie with a paragraph in another book (a book that does not correspond to this movie). 10 book experiment. In this experiment, we allow a clip in our 10 movie dataset (excluding the training movie) to match to paragraphs in the remaining 9 books (excluding the corresponding book). The results are in Fig. 12. Note that the top-scoring matches chosen from only a small set of books may not be too meaningful. 200 book experiment. We scale the experiment by randomly selecting 200 books from our BookCorpus. The results are in Fig. 15. One can see that by using many more books results in increasingly better “stories”. American Psycho American Psycho Harry Potter Figure 8: Examples of movie-book alignment. We use our model to align a movie to a book. Then for a chosen shot (which is a node in our CRF) we show the corresponding paragraph, plus one before and one after, in the book inferred by our model. On the left we show one (central) frame from the shot along with the subtitle sentence(s) that overlap with the shot. Some dialogs in the movie closely follow the book and thus help with the alignment. One Flew Over the Cuckoo’s Nest One Flew Over the Cuckoo’s Nest Shawshank Redemption Figure 9: Examples of movie-book alignment. We use our model to align a movie to a book. Then for a chosen shot (which is a node in our CRF) we show the corresponding paragraph, plus one before and one after, in the book inferred by our model. On the left we show one (central) frame from the shot along with the subtitle sentence(s) that overlap with the shot. Some dialogs in the movie closely follow the book and thus help with the alignment. The Firm The Firm The Firm Figure 10: Examples of movie-book alignment. We use our model to align a movie to a book. Then for a chosen shot (which is a node in our CRF) we show the corresponding paragraph, plus one before and one after, in the book inferred by our model. On the left we show one (central) frame from the shot along with the subtitle sentence(s) that overlap with the shot. Some dialogs in the movie closely follow the book and thus help with the alignment. The Green Mile The Green Mile The Road Figure 11: Examples of movie-book alignment. We use our model to align a movie to a book. Then for a chosen shot (which is a node in our CRF) we show the corresponding paragraph, plus one before and one after, in the book inferred by our model. On the left we show one (central) frame from the shot along with the subtitle sentence(s) that overlap with the shot. Some dialogs in the movie closely follow the book and thus help with the alignment. Figure 12: Examples of of borrowing paragraphs from other books – 10 book experiment. We show a few examples of top-scoring correspondences between a shot in a movie and a paragraph in a book that does not correspond to the movie. Note that by forcing the model to choose from another book, the top-scoring correspondences may still have a relatively low similarity. In this experiment, we did not enforce a global alignment over the full book – we use the similarity output by our contextual CNN. Figure 13: Examples of of borrowing paragraphs from other books – 10 book experiment. We show a few examples of top-scoring correspondences between a shot in a movie and a paragraph in a book that does not correspond to the movie. Note that by forcing the model to choose from another book, the top-scoring correspondences may still have a relatively low similarity. In this experiment, we did not enforce a global alignment over the full book – we use the similarity output by our contextual CNN. Figure 14: Examples of of borrowing paragraphs from other books – 200 book experiment. We show a few examples of top-scoring correspondences between a shot in a movie and a paragraph in a book that does not correspond to the movie. By scaling up the experiment (more books to choose from), our model gets increasingly more relevant “stories”. Figure 15: Examples of of borrowing paragraphs from other books – 200 book experiment. We show a few examples of top-scoring correspondences between a shot in a movie and a paragraph in a book that does not correspond to the movie. By scaling up the experiment (more books to choose from), our model gets increasingly more relevant “stories”. Bottom row: failed example. C. The CoCoBook We show more results for captioning CoCo images [18] with passages from the books. “ somewhere you ’ll never find it , ” owens sneered . if never meant five seconds , his claim was true . the little shit ’s gaze cut left , where a laptop sat on a coffee table . trey strode to it . owens ’ email program was open . seriously . its like a train crashing into another train . a train wreck . just something like that . i try to convince her . everyone was allowed to rest for the next twenty-four hours . that following evening : the elect , not their entourages , were called to a dining hall for supper with lady dolorous . a table that curved inward was laden with food and drink . the wall behind the table was windows with a view of the planet . girls in pink stood about and at attention . he had simply ... healed . brian watched his fellow passengers come aboard . a young woman with blonde hair was walking with a little girl in dark glasses . the little girl ’s hand was on the blonde ’s elbow . the woman murmured to her charge , the girl looked immediately toward the sound of her voice , and brian understood she was blind - it was something in the gesture of the head . this was a beautiful miniature reproduction of a real london town house , and when jessamine touched it , tessa saw that the front of it swung open on tiny hinges . tessa caught her breath . there were beautiful tiny rooms perfectly decorated with miniature furniture , everything built to scale , from the little wooden chairs with needlepoint cushions to the cast-iron stove in the kitchen . there were small dolls , too , with china heads , and real little oil paintings on the walls . “ this was my house . ” if he had been nearby he would have dragged her out of the room by her hair and strangled her . during lunch break she went with a group back to the encampment . out of view of the house , under a stand of towering trees , several tents were sitting in a field of mud . the rain the night before had washed the world , but here it had made a mess of things . a few women fired up a camp stove and put on rice and lentils . then a frightened yell . “ hang on ! ” suddenly , jake was flying through the air . nefertiti became airborne , too . he screamed , not knowing what was happening-then he splashed into a pool of water . grabbing his wristwatch off the bedside table he checked the time , grimacing when he saw that it was just after two in the afternoon . jeanne louise should n’t be up yet . stifling a yawn , he slid out of bed and made his way to the en suite bathroom for a shower . twenty minutes later paul was showered , dressed , and had brushed his teeth and hair . feeling somewhat alive now , he made his way out of his and jeanne louise ’s room , pausing to look in on livy as he passed . she cried . quentin put a heavy , warm , calming hand on her thigh , saying , “ he should be sober by then . ” a cell phone rang . he pulled his from his back pocket , glanced at it , then used the remote to turn the tv to the channel that showed the feed from the camera at the security gate . “ oh , it ’s rachel . ” now however she was out of his shot . he had missed it completely until he had ended up on the ground with his shotgun . an old clock hung on the wall near the door . the was obviously broken , the small red hand ticking the same second away over and over again . morgan squeezed the trigger and pellets ripped out of their package , bounced down the barrel , flew through the air and ripped into the old clock tearing it in two before it smashed to the ground . a man sat in a chair , facing the wall opposite of me . it nearly startled me when i first saw him , and made a bit of a squeak , but he did nothing . he had dark gray hair , a black suit and pants , and a gray and blue striped tie . s-sir ? i said . its been years since we last played together , but as i recall , he was rather weak at the net . or was it his serving ? all i know is he plays tennis much better than he plays cricket . perhaps , mr brearly , frances eventually replied , we should wait until we actually start playing . then we can ascertain our oppositions faults , and make a plan based on the new information . since it was the middle of summer , there were candles in the fireplace instead of a fire . but it still cast a romantic glow over the room . there were candles on the mantle and on a table set up in the corner with flowers . as she looked around , her eyes instinctively turned to find max who was behind a bar opening a bottle of champagne . the doors were closed quietly behind her and her mouth felt dry as she looked across the room at the man who had haunted her dreams for so long . the open doorway of another house provided a view of an ancient game of tiles . it wasnt the game that held reddings attention . it was the four elderly people who sat around a table playing the game . they were well beyond their productive years and the canal township had probably been their whole lives . redding and lin ming stepped away from the doorway right into the path of a wooden pushcart . along with the fish , howard had given them some other picnic treats that had spoiled ... mushrooms in cream sauce , rotted greens . the bats and temp were only eating from the river now , but the remaining picnic food was running low . there were a few loaves of stale bread , some cheese , some dried vegetables , and a couple of cakes . gregor looked over the supplies and thought about boots wailing for food and water in the jungle . it had been unbearable . he felt the first stirrings of fear mixing with his anger . a light flicked on in the room and eric jerked , blinking for a minute at the brightness before the images focused . there was a tall , thin man standing over a mannequin . he looked like he was assembling it , since its leg was on the ground next to the man and its arm was in two pieces farther away . then the mannequin ’s head turned . References [1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015. 4 [2] K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP, 2014. 4 [3] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. 4 [4] T. Cour, C. Jordan, E. Miltsakaki, and B. Taskar. Movie/script: Alignment and parsing of video and text transcription. In ECCV, 2008. 2 [5] M. Everingham, J. Sivic, and A. Zisserman. “Hello! My name is... Buffy” – Automatic Naming of Characters in TV Video. BMVC, pages 899–908, 2006. 2 [6] A. Farhadi, M. Hejrati, M. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences for images. In ECCV, 2010. 2 [7] S. Fidler, A. Sharma, and R. Urtasun. A sentence is worth a thousand pixels. In CVPR, 2013. 2 [8] A. Gupta and L. Davis. Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In ECCV, 2008. 1 [9] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 4 [10] N. Kalchbrenner and P. Blunsom. Recurrent continuous translation models. In EMNLP, pages 1700–1709, 2013. 4 [11] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. 1, 2 [12] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 5 [13] R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. CoRR, abs/1411.2539, 2014. 1, 2, 3, 5, 9, 10 [14] R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel, A. Torralba, R. Urtasun, and S. Fidler. Skip-Thought Vectors. In Arxiv, 2015. 3, 4 [15] C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to-image coreference. In CVPR, 2014. 1, 2 [16] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. Berg, and T. Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR, 2011. 2 [17] D. Lin, S. Fidler, C. Kong, and R. Urtasun. Visual Semantic Search: Retrieving Videos via Complex Textual Queries. CVPR, pages 2657–2664, 2014. 1, 2 [18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. 2014. 1, 19 [19] X. Lin and D. Parikh. Don’t just listen, use your imagination: Leveraging visual common sense for non-visual tasks. In CVPR, 2015. 1 [20] M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. 1 [21] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. In arXiv:1410.1090, 2014. 1, 2 [22] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. 4 [23] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. BLEU: a method for automatic evaluation of machine translation. In ACL, pages 311–318, 2002. 6 [24] H. Pirsiavash, C. Vondrick, and A. Torralba. Inferring the why in images. arXiv.org, jun 2014. 2 [25] V. Ramanathan, A. Joulin, P. Liang, and L. Fei-Fei. Linking People in Videos with “Their” Names Using Coreference Resolution. In ECCV, pages 95–110. 2014. 2 [26] V. Ramanathan, P. Liang, and L. Fei-Fei. Video event understanding using natural language descriptions. In ICCV, 2013. 1 [27] A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele. A dataset for movie description. In CVPR, 2015. 2, 5 [28] P. Sankar, C. V. Jawahar, and A. Zisserman. Subtitle-free Movie to Script Alignment. In BMVC, 2009. 2 [29] A. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun. Efficient Structured Prediction with Latent Variables for General Graphical Models. In ICML, 2012. 6 [30] J. Sivic, M. Everingham, and A. Zisserman. “Who are you?” - Learning person specific classifiers from video. CVPR, pages 1145–1152, 2009. 2 [31] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. 4 [32] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. 5 [33] M. Tapaswi, M. Bauml, and R. Stiefelhagen. Book2Movie: Aligning Video scenes with Book chapters. In CVPR, 2015. 2 [34] M. Tapaswi, M. Buml, and R. Stiefelhagen. Aligning Plot Synopses to Videos for Story-based Retrieval. IJMIR, 4:3– 16, 2015. 1, 2, 6 [35] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. J. Mooney, and K. Saenko. Translating Videos to Natural Language Using Deep Recurrent Neural Networks. CoRR abs/1312.6229, cs.CV, 2014. 1, 2 [36] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In arXiv:1411.4555, 2014. 1, 2 [37] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In arXiv:1502.03044, 2015. 2 [38] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning Deep Features for Scene Recognition using Places Database. In NIPS, 2014. 5, 7 Deep Networks for Image Super-Resolution with Sparse Prior Zhaowen Wang†‡ Ding Liu† Jianchao Yang§ Wei Han† Thomas Huang† † Beckman Institute, University of Illinois at Urbana-Champaign, Urbana, IL ‡ Adobe Research, San Jose, CA § Snapchat, Venice, CA arXiv:1507.08905v4 [cs.CV] 15 Oct 2015 ‡ zhawang@adobe.com † {dingliu2, weihan3, huang}@ifp.uiuc.edu Abstract § jcyangenator@gmail.com external databases [36, 35], or recovered from similar examples in the LR image itself at different locations and across different scales [13, 12, 32]. A comprehensive review of more SR methods can be found in [33]. More recently, inspired by the great success achieved by deep learning [18, 27, 30] in other computer vision tasks, people begin to use neural networks with deep architecture for image SR. Multiple layers of collaborative autoencoders are stacked together in [6] for robust matching of self-similar patches. Deep convolutional neural networks (CNN) [8] and deconvolutional networks [25] are designed that directly learn the non-linear mapping from LR space to HR space in a way similar to coupled sparse coding [35]. As these deep networks allow end-to-end training of all the model components between LR input and HR output, significant improvements have been observed over their shadow counterparts. The networks in [6, 8] are built with generic architectures, which means all their knowledge about SR is learned from training data. On the other hand, people’s domain expertise for the SR problem, such as natural image prior and image degradation model, is largely ignored in deep learning based approaches. It is then worthy to investigate whether domain expertise can be used to design better deep model architectures, or whether deep learning can be leveraged to improve the quality of handcrafted models. In this paper, we extend the conventional sparse coding model [36] using several key ideas from deep learning, and show that domain expertise is complementary to large learning capacity in further improving SR performance. First, based on the learned iterative shrinkage and thresholding algorithm (LISTA) [14], we implement a feed-forward neural network whose layers strictly correspond to each step in the processing flow of sparse coding based image SR. In this way, the sparse representation prior is effectively encoded in our network structure; at the same time, all the components of sparse coding can be trained jointly through back-propagation. This simple model, which is named sparse coding based network (SCN), achieves notable improvement over the generic CNN model [8] in terms Deep learning techniques have been successfully applied in many areas of computer vision, including low-level image restoration problems. For image super-resolution, several models based on deep neural networks have been recently proposed and attained superior performance that overshadows all previous handcrafted models. The question then arises whether large-capacity and data-driven models have become the dominant solution to the ill-posed superresolution problem. In this paper, we argue that domain expertise represented by the conventional sparse coding model is still valuable, and it can be combined with the key ingredients of deep learning to achieve further improved results. We show that a sparse coding model particularly designed for super-resolution can be incarnated as a neural network, and trained in a cascaded structure from end to end. The interpretation of the network based on sparse coding leads to much more efficient and effective training, as well as a reduced model size. Our model is evaluated on a wide range of images, and shows clear advantage over existing state-of-the-art methods in terms of both restoration accuracy and human subjective quality. 1. Introduction Single image super-resolution (SR) aims at obtaining a high-resolution (HR) image from a low-resolution (LR) input image by inferring all the missing high frequency contents. With the known variables in LR images greatly outnumbered by the unknowns in HR images, SR is a highly ill-posed problem and the current techniques are far from being satisfactory for many real applications [2, 21]. To regularize the solution of SR, people have exploited various priors of natural images. Analytical priors, such as bicubic interpolation, work well for smooth regions; while image models based on statistics of edges [11] and gradients [17, 1] can recover sharper structures. In the patch-based SR methods, HR patch candidates are represented as the sparse linear combination of dictionary atoms trained from 1 of both recovery accuracy and human perception, and yet has a compact model size. Moreover, with the correct understanding of each layer’s physical meaning, we have a more principled way to initialize the parameters of SCN, which helps to improve optimization speed and quality. A single network is only able to perform image SR by a particular scaling factor. In [8], different networks are trained for different scaling factors. In this paper, we also propose a cascade of multiple SCNs to achieve SR for arbitrary factors. This simple approach, motivated by the self-similarity based SR approach [13], not only increases the scaling flexibility of our model, but also reduces artifacts for large scaling factors. The cascade of SCNs (CSCN) can also benefit from the end-to-end training of deep network with a specially designed multi-scale cost function. In short, the contributions of this paper include: • combine the domain expertise of sparse coding and the merits of deep learning to achieve better SR performance with faster training and smaller model size; • use network cascading for large and arbitrary scaling factors; • conduct a subjective evaluation on several recent stateof-the-art methods. In the following, we will first review related work in Sec. 2. The SCN and CSCN models are introduced in Sec. 3 and Sec. 4, with implementation details in Sec. 5. Extensive experimental results are reported in Sec. 6, and conclusions are drawn in Sec. 7. 2. Related Work 2.1. Image SR Using Sparse Coding The sparse representation based SR method [36] models the transform from each local patch y ∈ Rmy in the bicubic-upscaled LR image to the corresponding patch x ∈ Rmx in the HR image. The dimension my is not necessarily the same as mx when image features other than raw pixel is used to represent patch y. It is assumed that the LR(HR) patch y(x) can be represented with respect to an overcomplete dictionary D y (D x ) using some sparse linear coefficients αy (αx ) ∈ Rn , which are known as sparse code. Since the degradation process from x to y is nearly linear, the patch pair can share the same sparse code αy = αx = α if the dictionaries D y and D x are defined properly. Therefore, for an input LR patch y, the HR patch can be recovered as x = D x α, s.t. α = arg min ky − D y zk22 + λkzk1 , (1) z where k · k1 denotes the `1 norm which is convex and sparsity-inducing, and λ is a regularization coefficient. The dictionary pair (D y , D x ) can be learned alternatively with the inference of training patches’ sparse codes in their joint space [36] or through bi-level optimization [35]. Figure 1. A LISTA network [14] with 2 time-unfolded recurrent stages, whose output α is an approximation of the sparse code of input signal y. The linear weights W , S and the shrinkage thresholds θ are learned from data. 2.2. Network Implementation of Sparse Coding There is an intimate connection between sparse coding and neural network, which has been well studied in [16, 14]. A feed-forward neural network as illustrated in Fig. 1 is proposed in [14] to efficiently approximate the sparse code α of input signal y as it would be obtained by solving (1) for a given dictionary D y . The network has a finite number of recurrent stages, each of which updates the intermediate sparse code according to z k+1 = hθ (W y + Sz k ), (2) where hθ is an element-wise shrinkage function defined as [hθ (a)]i = sign(ai )(|ai | − θi )+ with positive thresholds θ. Different from the iterative shrinkage and thresholding algorithm (ISTA) [7, 26] which finds an analytical relationship between network parameters (weights W , S and thresholds θ) and sparse coding parameters (D y and λ), the authors of [14] learn all the network parameters from training data using a back-propagation algorithm called learned ISTA (LISTA). In this way, a good approximation of the underlying sparse code can be obtained within a fixed number of recurrent stages. 3. Sparse Coding based Network for Image SR Given the fact that sparse coding can be effectively implemented with a LISTA network, it is straightforward to build a multi-layer neural network that mimics the processing flow of the sparse coding based SR method [36]. Same as most patch-based SR methods, our sparse coding based network (SCN) takes the bicubic-upscaled LR image I y as input, and outputs the full HR image I x . Fig. 2 shows the main network structure, and each of the layers is described in the following. The input image I y first goes through a convolutional layer H which extracts feature for each LR patch. There are my filters of spatial size sy ×sy in this layer, so that our input patch size is sy ×sy and its feature representation y has my dimensions. Each LR patch y is then fed into a LISTA network with a finite number of k recurrent stages to obtain its sparse code α ∈ Rn . Each stage of LISTA consists of two linear Figure 2. Top left: the proposed SCN model with a patch extraction layer H, a LISTA sub-network for sparse coding (with k recurrent stages denoted by the dashed box), a HR patch recovery layer D x , and a patch combination layer G. Top right: a neuron with an adjustable threshold decomposed into two linear scaling layers and a unit-threshold neuron. Bottom: the SCN re-organized with unitthreshold neurons and adjacent linear layers merged together in the gray boxes. layers parameterized by W ∈ Rn×my and S ∈ Rn×n , and a nonlinear neuron layer with activation function hθ . The activation thresholds θ ∈ Rn are also to be updated during training, which complicates the learning algorithm. To restrict all the tunable parameters in our linear layers, we do a simple trick to rewrite the activation function as [hθ (a)]i = sign(ai )θi (|ai |/θi − 1)+ = θi h1 (ai /θi ). (3) Eq. (3) indicates the original neuron with an adjustable threshold can be decomposed into two linear scaling layers and a unit-threshold neuron, as shown in the top-right of Fig. 2. The weights of the two scaling layers are diagonal matrices defined by θ and its element-wise reciprocal, respectively. The sparse code α is then multiplied with HR dictionary D x ∈ Rmx ×n in the next linear layer, reconstructing HR patch x of size sx ×sx = mx . In the final layer G, all the recovered patches are put back to the corresponding positions in the HR image I x . This is realized via a convolutional filter of mx channels with spatial size sg ×sg . The size sg is determined as the number of neighboring patches that overlap with the same pixel in each spatial direction. The filter will assign appropriate weights to the overlapped recoveries from different patches and take their weighted average as the final prediction in I x . As illustrated in the bottom of Fig. 2, after some simple reorganizations of the layer connections, the network described above has some adjacent linear layers which can be merged into a single layer. This helps to reduce the computation load as well as redundant parameters in the network. The layers H and G are not merged because we apply additional nonlinear normalization operations on patches y and x, which will be detailed in Sec. 5. Thus, there are totally 5 trainable layers in our network: 2 convolutional layers H and G, and 3 linear layers shown as gray boxes in Fig. 2. The k recurrent layers share the same weights and are therefore conceptually regarded as one. Note that all the linear layers are actually implemented as convolutional layers applied on each patch with filter spatial size of 1×1, a structure similar to the network in network [20]. Also note that all these layers have only weights but no biases (zero biases). Mean square error (MSE) is employed as the cost function to train the network, and our optimization objective can be expressed as X (i) 2 (4) min kSCN (I (i) y ; Θ) − I x k2 , Θ i (i) where I (i) y and I x are the i-th pair of LR/HR training data, and SCN (I y ; Θ) denotes the HR image for I y predicted using the SCN model with parameter set Θ. All the parameters are optimized through the standard back-propagation algorithm. Although it is possible to use other cost terms that are more correlated with human visual perception than MSE, our experimental results show that simply minimizing MSE leads to improvement in subjective quality. Advantages over Previous Models The construction of our SCN follows exactly each step in the sparse coding based SR method [36]. If the network parameters are set according to the dictionaries learned in [36], it can reproduce almost the same results. However, after training, SCN learns a more complex regression function and can no longer be converted to an equivalent sparse coding model. The advantage of SCN comes from its ability to jointly optimize all the layer parameters from end to end; while in [36] some variables are manually designed and some are optimized individually by fixing all the others. Technically, our network is also a CNN and it has similar layers as the CNN model proposed in [8] for patch extraction and reconstruction. The key difference is that we have a LISTA sub-network specifically designed to enforce sparse representation prior; while in [8] a generic rectified linear unit (ReLU) [24] is used for nonlinear mapping. Since SCN is designed based on our domain knowledge in sparse coding, we are able to obtain a better interpretation of the filter responses and have a better way to initialize the filter parameters in training. We will see in the experiments that all these contribute to better SR results, faster training speed and smaller model size than a vanilla CNN. (a) LR image 4. Network Cascade for Scalable SR Like most SR models learned from external training examples, the SCN discussed previously can only upscale images by a fixed factor. A separate model needs to be trained for each scaling factor to achieve the best performance, which limits the flexibility and scalability in practical use. One way to overcome this difficulty is to repeatedly enlarge the image by a fixed scale until the resulting HR image reaches a desired size. This practice is commonly adopted in the self-similarity based methods [13, 12, 6], but is not so popular in other cases for the fear of error accumulation during repetitive upscaling. In our case, however, it is observed that a cascade of SCNs (CSCN) trained for small scaling factors can generate even better SR results than a single SCN trained for a large scaling factor, especially when the target scaling factor is large (greater than 2). This is illustrated by the example in Fig. 3. Here an input image is magnified by ×4 times in two ways: with a single SCN×4 model through the processing flow (a) → (b) → (d); and with a cascade of two SCN×2 models through (a) → (c) → (e). It can be seen that the input to the second cascaded SCN×2 in (c) is already sharper and contains less artifacts than the bicubic×4 input to the single SCN×4 in (b), which naturally leads to the better final result in (e) than the one in (d). Therefore, each SCN in the cascade serves as a “relaying station” which progressively recovers some useful information lost in bicubic interpolation and compensates for the distortion aggregated from previous stages. The CSCN is also a deep network, in which the output of each SCN is connected to the input of the next SCN with bicubic interpolation in the between. To construct the cascade, besides stacking several SCNs trained individually with respect to (4), we can also optimize all of them jointly as shown in Fig. 4. Without loss of generality, we assume each SCN in the cascade has the same scaling factor s. Let I 0 denote the input image of original size, and Iˆj (j>0) denote the output image of the j-th SCN upscaled by a total of ×sj times. Each Iˆj can be compared with its associated ground truth image I j according to the MSE cost, leading to a multi-scale objective function: 2 XX (i) (i) min SCN (Iˆj−1 ↑s; Θj ) − I j , (5) {Θj } i j 2 where i denotes the data index, and j denotes the SCN (b) bicubic×4 (28.52) (c) SCN×2 & bicubic×2 (30.27) (d) SCN×4 (30.22) (e) SCN×2 & SCN×2 (30.72) Figure 3. SR results for the “Lena” image upscaled by 4 times. (a) → (b) → (d) represents the processing flow with a single SCN×4 model. (a) → (c) → (e) represents the processing flow with two cascaded SCN×2 models. PSNR is given in parentheses. Figure 4. Training cascade of SCNs with multi-scale objectives. index. I↑s is the bicubic interpolated image of I by a factor of s. This multi-scale objective function makes full use of the supervision information in all scales, sharing a similar idea as heterogeneous networks [19, 5]. All the layer parameters {Θj } in (5) could be optimized from end to end by back-propagation. We use a greedy algorithm here to train each SCN sequentially from the beginning of the cascade so that we do not need to care about the gradient of bicubic layers. Applying back-propagation through a bicubic layer or its trainable surrogate will be considered in future work. 5. Implementation Details We determine the number of nodes in each layer of our SCN mainly according to the corresponding settings used in sparse coding [35]. Unless otherwise stated, we use input LR patch size sy =9, LR feature dimension my =100, dictionary size n=128, output HR patch size sx =5, and patch aggregation filter size sg =5. All the convolution layers have a stride of 1. Each LR patch y is normalized by its mean and variance, and the same mean and variance are used to restore the final HR patch x. We crop 56×56 regions from each image to obtain fixed-sized input samples to the network, which produces outputs of size 44×44. To reduce the number of parameters, we implement the LR patch extraction layer H as the combination of two layers: the first layer has 4 trainable filters each of which is shifted to 25 fixed positions by the second layer. Similarly, the patch combination layer G is also split into a fixed layer which aligns pixels in overlapping patches and a trainable layer whose weights are used to combine overlapping pixels. In this way, the number of parameters in these two layers are reduced by more than an order, and there is no observable loss in performance. We employ a standard stochastic gradient descent algorithm to train our networks with mini-batch size of 64. Based on the understanding of each layer’s role in sparse coding, we use Harr-like gradient filters to initialize layer H, and use uniform weights to initialize layer G. All the remaining three linear layers are related to the dictionary pair (D x , D y ) in sparse coding. To initialize them, we first randomly set D x and D y with Gaussian noise, and then find the corresponding layer weights as in ISTA [7]: w1 = C·D Ty , w2 = I−D Ty D y , w3 = (CL)−1 ·D x (6) where w1 , w2 and w3 denote the weights of the three subsequent layers after layer H. L is the upper bound on the largest eigenvalue of D Ty D y , and C is the threshold value before normalization. We empirically set L=C=5. The proposed models are all trained using the CUDA ConvNet package [18] on a workstation with 12 Intel Xeon 2.67GHz CPUs and 1 GTX680 GPU. Training a SCN usually takes less than one day. Note that this package is customized for classification networks, and its efficiency can be further optimized for our SCN model. In testing, to make the entire image covered by output samples, we crop input samples with overlap and extend the boundary of original image by reflection. Note we shave the image border in the same way as [8] for objective evaluations to ensure fair comparison. Only the luminance channel is processed with our method, and bicubic interpolation is applied to the chrominance channels. To achieve arbitrary upscaling factors using CSCN, we upscale an image by ×2 times repeatedly until it is at least as large as the desired size. Then a bicubic interpolation is used to downscale it to the target resolution if necessary. When reporting our best results in Sec. 6.2, we also use the multi-view testing strategy commonly employed in image classification. For patch-based image SR, multi-view Figure 5. The four learned filters in the first layer H. Figure 6. The PSNR change for ×2 SR on Set5 during training using different methods: SCN; SCN with random initialization; CNN. The horizontal dash lines show the benchmarks of bicubic interpolation and sparse coding (SC). testing is implicitly used when predictions from multiple overlapping patches are averaged. Here, besides sampling overlapping patches, we also add more views by flipping and transposing the patch. Such strategy is found to improve SR performance for general algorithms at the sheer cost of computation. 6. Experiments We evaluate and compare the performance of our models using the same data and protocols as in [28], which are commonly adopted in SR literature. All our models are learned from a training set with 91 images, and tested on Set5 [3], Set14 [37] and BSD100 [23] which contain 5, 14 and 100 images respectively. We have also trained on a different larger data set, and observe little performance change (less than 0.1dB). The original images are downsized by bicubic interpolation to generate LR-HR image pairs for both training and evaluation. The training data are augmented with translation, rotation and scaling. 6.1. Algorithm Analysis We first visualize the four filters learned in the first layer H in Fig. 5. The filter patterns do not change much from the initial first and second order gradient operators. Some additional small coefficients are introduced in a highly structured form that capture richer high frequency details. The performance of several networks during training is measured on Set5 in Fig. 6. Our SCN improves significantly over sparse coding (SC) [35], as it leverages data more effectively with end-to-end training. The SCN initialized according to (6) can converge faster and better than the artifacts quickly get amplified through many repetitive upscalings. Therefore, we use SCN×2 as the default building block for CSCN, and drop the notation ×2 when there is no ambiguity. The last row in Table 1 shows that a CSCN trained using the multi-scale objective in (5) can further improve the SR results for scaling factors 3 and 4, as the second SCN in the cascade is trained to be robust to the artifacts generated by the first one. 6.2. Comparison with State of the Arts Figure 7. PSNR for ×2 SR on Set5 using SCN and CNN with various network sizes. Table 1. PSNR of different network cascading schemes on Set5, evaluated for different scaling factors in each column. upscale factor SCN×1.5 SCN×2 SCN×3 SCN×4 CSCN ×1.5 40.14 40.15 39.88 39.69 40.15 ×2 36.41 36.93 36.76 36.54 36.93 ×3 30.33 32.99 32.87 32.76 33.10 ×4 29.02 30.70 30.63 30.55 30.86 same model with random initialization, which indicates that the understanding of SCN based on sparse coding can help its optimization. We also train a CNN model [8] of the same size as SCN, but find its convergence speed much slower. It is reported in [8] that training a CNN takes 8×108 back-propagations (equivalent to 12.5×106 minibatches here). To achieve the same performance as CNN, our SCN requires less than 1% back-propagations. The network size of SCN is mainly determined by the dictionary size n. Besides the default value n=128, we have tried other sizes and plot their performance versus the number of network parameters in Fig. 7. The PSNR of SCN does not drop too much as n decreases from 128 to 64, but the model size and computation time can be reduced significantly. Fig. 7 also shows the performance of CNN with various sizes. Our smallest SCN can achieve higher PSNR than the largest model (CNN-L) in [9] while only using about 20% parameters. Different numbers of recurrent stages k have been tested for SCN, and we find increasing k from 1 to 3 only improves performance by less than 0.1dB. As a tradeoff between speed and accuracy, we use k=1 throughout the paper. In Table 1, different network cascade structures (in each row) are compared at different scaling factors (in each column). SCN×a denotes the simple cascade of SCN with fixed scaling factor a, where an individually trained SCN is applied one or more times for scaling factors other than a. It is observed that SCN×2 can perform as well as the scale-specific model for small scaling factor (1.5), and much better for large scaling factors (3 and 4). Note that the cascade of SCN×1.5 does not lead to good results since We compare the proposed CSCN with other recent SR methods on all the images in Set5, Set14 and BSD100 for different upscaling factors. Table 2 shows the PSNR and structural similarity (SSIM) [31] for adjusted anchored neighborhood regression (A+) [29], CNN [8], CNN trained with larger model size and more data (CNN-L) [9], the proposed CSCN, and CSCN with our multi-view testing (CSCN-MV). We do not list other methods [35, 28, 37, 17, 15] whose performance is worse than A+ or CNN-L. It can be seen from Table 2 that CSCN performs consistently better than all previous methods in both PSNR and SSIM, and with multi-view testing the results can be further improved. CNN-L improves over CNN by increasing model parameters and training data. However, it is still not as good as CSCN which is trained with a much smaller size and on a much smaller data set. Clearly, the better model structure of CSCN makes it less dependent on model capacity and training data in improving performance. Our models are generally more advantageous for large scaling factors due to the cascade structure. The visual qualities of the SR results generated by sparse coding (SC) [35], CNN and CSCN are compared in Fig. 8. Our approach produces image patterns with shaper boundaries and richer textures, and is free of the ringing artifacts observable in the other two methods. Fig. 9 shows the SR results on the “chip” image compared among more methods including the self-example based method (SE) [12] and the deep network cascade (DNC) [6]. SE and DNC can generate very sharp edges on this image, but also introduce artifacts and blurs on corners and fine structures due to the lack of self-similar patches. On the contrary, the CSCN method recovers all the structures of the characters without any distortion. We also compare CSCN with other sparse coding extensions [22, 10, 38], and consider the blurring effect introduced in downscaling. A PSNR gain of 0.3∼1.6dB is achieved by CSCN in general. Experiment details and source codes are available online1 . 6.3. Subjective Evaluation We conducted a subjective evaluation of SR results for several methods including bicubic, SC [35], SE [12], 1 www.ifp.illinois.edu/ ˜dingliu2/iccv15 Table 2. PSNR (SSIM) comparison on three test data sets among different methods. Red indicates the best and blue indicates the second best performance. The performance gain of our best model over all the others’ best is shown in the last row. Data Set Set5 Set14 BSD100 Upscaling ×2 ×3 ×4 ×2 ×3 ×4 ×2 ×3 ×4 A+ [29] CNN [8] CNN-L [9] CSCN CSCN-MV 32.59 (0.9088) 32.39 (0.9033) 32.75 (0.9090) 33.10 (0.9144) 33.26 (0.9167) 30.29 (0.8603) 30.09 (0.8530) 30.49 (0.8628) 30.86 (0.8732) 31.04 (0.8775) 32.28 (0.9056) 32.18 (0.9039) 32.45 (0.9067) 32.56 (0.9074) 32.71 (0.9095) 29.13 (0.8188) 29.00 (0.8145) 29.30 (0.8215) 29.41 (0.8238) 29.55 (0.8271) 27.33 (0.7491) 27.20 (0.7413) 27.50 (0.7513) 27.64 (0.7578) 27.76 (0.7620) 30.78 (0.8773) 31.11 (0.8835) 31.36 (0.8879) 31.40 (0.8884) 31.54 (0.8908) 28.18 (0.7808) 28.20 (0.7794) 28.41 (0.7863) 28.50 (0.7885) 28.58 (0.7910) 26.77 (0.7085) 26.70 (0.7018) 26.90 (0.7103) 27.03 (0.7161) 27.11 (0.7191) 0.48 (0.0023) 0.51 (0.0077) 0.55 (0.0147) 0.26 (0.0028) 0.25 (0.0056) 0.26 (0.0107) 0.18 (0.0029) 0.17 (0.0047) 0.21 (0.0088) CSCN CNN SC Our Improvement 36.55 (0.9544) 36.34 (0.9521) 36.66 (0.9542) 36.93 (0.9552) 37.14 (0.9567) Figure 8. SR results given by SC [35] (first row), CNN [8] (second row) and our CSCN (third row). Images from left to right: the “monarch” image upscaled by ×3; the “zebra” image upscaled by ×3; the “comic” image upscaled by ×3. self-example regression (SER) [34], CNN [8] and CSCN. Ground truth HR images are also included when they are available as references. Each of the participants in the evaluation is shown a set of HR image pairs, which are (a) bicubic (b) SE [12] (c) SC [35] (d) DNC [6] (e) CNN [8] Figure 9. The “chip” image upscaled by ×4 times using different methods. (f) CSCN Figure 10. Subjective SR quality scores for different methods including bicubic, SC [35], SE [12], SER [34], CNN [8] and the proposed CSCN. The score for ground truth result is 1. upscaled from the same LR images using two randomly selected methods. For each pair, the subject needs to decide which one is better in terms of perceptual quality. We have a total of 270 participants giving 720 pairwise comparisons over 6 images with different scaling factors. Not every participant completed all the comparisons but their partial responses are still useful. All the evaluation results can be summarized into a 7×7 winning matrix for 7 methods (including ground truth), based on which we fit a Bradley-Terry [4] model to estimate the subjective score for each method so that they can be ranked. Fig. 10 shows the estimated scores for the 6 SR methods in our evaluation, with the score for ground truth method normalized to 1. As expected, all the SR methods have much lower scores than ground truth, showing the great challenge in SR problem. The bicubic interpolation is significantly worse than other SR methods. The proposed CSCN method outperforms other previous state-of-the-art methods by a large margin, demonstrating its superior visual quality. It should be noted that the visual difference between some image pairs is very subtle. Nevertheless, the human subjects are able to perceive such difference when seeing the two images side by side, and therefore make consistent ratings. The CNN model becomes less competitive in the subjective evaluation than it is in PSNR comparison. This indicates that the visually appealing image appearance produced by CSCN should be attributed to the regularization from sparse representation, which can not be easily learned by merely minimizing reconstruction error as in CNN. 7. Conclusions We propose a new model for image SR by combining the strengths of sparse coding and deep network, and make considerable improvement over existing deep and shallow SR models both quantitatively and qualitatively. Besides producing good SR results, the domain knowledge in the form of sparse coding can also benefit training speed and model compactness. Furthermore, we propose a cascaded network for better flexibility in scaling factors as well as more robustness to artifacts. In future work, we will apply the SCN model to other problems where sparse coding can be useful. The interaction between deep networks for low-level and high-level vision tasks will also be explored. References [1] H. A. Aly and E. Dubois. Image up-sampling using totalvariation regularization with a new observation model. IEEE TIP, 14(10):1647–1659, 2005. 1 [2] S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE TPAMI, 24(9):1167–1183, 2002. 1 [3] M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. A. Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 5 [4] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pages 324–345, 1952. 8 [5] S. Chang, W. Han, J. Tang, G.-J. Qi, C. C. Aggarwal, and T. S. Huang. Heterogeneous network embedding via deep architectures. In ACM SIGKDD. ACM, 2015. 4 [6] Z. Cui, H. Chang, S. Shan, B. Zhong, and X. Chen. Deep network cascade for image super-resolution. In ECCV, pages 49–64, 2014. 1, 4, 6, 8 [7] I. Daubechies, M. Defrise, and C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57(11):1413–1457, 2004. 2, 5 [8] C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In ECCV, pages 184–199, 2014. 1, 2, 3, 5, 6, 7, 8 [9] C. Dong, C. C. Loy, K. He, and X. Tang. Image superresolution using deep convolutional networks. IEEE TPAMI, 2015. 6, 7 [10] W. Dong, L. Zhang, and G. Shi. Centralized sparse representation for image restoration. In ICCV, pages 1259–1266, 2011. 6 [11] R. Fattal. Image upsampling via imposed edge statistics. In ACM Transactions on Graphics, volume 26:3, page 95, 2007. 1 [12] G. Freedman and R. Fattal. Image and video upscaling from local self-examples. ACM Transactions on Graphics, 30(2):12, 2011. 1, 4, 6, 8 [13] D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In ICCV, 2009. 1, 2, 4 [14] K. Gregor and Y. LeCun. Learning fast approximations of sparse coding. In ICML, 2010. 1, 2 [15] J.-B. Huang, A. Singh, and N. Ahuja. Single image superresolution from transformed self-exemplars. In CVPR, 2015. 6 [16] K. Kavukcuoglu, M. Ranzato, and Y. LeCun. Fast inference in sparse coding algorithms with applications to object recognition. arXiv preprint arXiv:1010.3467, 2010. 2 [17] K. I. Kim and Y. Kwon. Single-image super-resolution using sparse regression and natural image prior. IEEE TPAMI, 32(6):1127–1133, 2010. 1, 6 [18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, pages 1097–1105, 2012. 1, 5 [19] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeplysupervised nets. arXiv preprint arXiv:1409.5185, 2014. 4 [20] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013. 3 [21] Z. Lin and H.-Y. Shum. Fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE TPAMI, 26(1):83–97, 2004. 1 [22] X. Lu, H. Yuan, P. Yan, Y. Yuan, and X. Li. Geometry constrained sparse coding for single image super-resolution. In CVPR, pages 1648–1655, 2012. 6 [23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, volume 2, pages 416–423, July 2001. 5 [24] V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, pages 807–814, 2010. 4 [25] C. Osendorfer, H. Soyer, and P. van der Smagt. Image super-resolution with fast approximate convolutional sparse coding. In Neural Information Processing, pages 250–257. Springer, 2014. 1 [26] C. J. Rozell, D. H. Johnson, R. G. Baraniuk, and B. A. Olshausen. Sparse coding via thresholding and local competition in neural circuits. Neural Computation, 20(10):2526– 2563, 2008. 2 [27] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013. 1 [28] R. Timofte, V. De, and L. V. Gool. Anchored neighborhood regression for fast example-based super-resolution. In ICCV, pages 1920–1927, 2013. 5, 6 [29] R. Timofte, V. De Smet, and L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV, 2014. 6, 7 [30] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, pages 1096–1103, 2008. 1 [31] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600–612, 2004. 6 [32] Z. Wang, Y. Yang, Z. Wang, S. Chang, J. Yang, and T. S. Huang. Learning super-resolution jointly from external and internal examples. IEEE TIP, 24(11):4359–4371, 2015. 1 [33] C.-Y. Yang, C. Ma, and M.-H. Yang. Single-image superresolution: a benchmark. In ECCV, pages 372–386, 2014. 1 [34] J. Yang, Z. Lin, and S. Cohen. Fast image super-resolution based on in-place example regression. In CVPR, pages 1059–1066. IEEE, 2013. 7, 8 [35] J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang. Coupled dictionary training for image super-resolution. IEEE TIP, 21(8):3467–3478, 2012. 1, 2, 4, 5, 6, 7, 8 [36] J. Yang, J. Wright, T. Huang, and Y. Ma. Image superresolution via sparse representation. IEEE TIP, 19(11):1–8, 2010. 1, 2, 3 [37] R. Zeyde, M. Elad, and M. Protter. On single image scale-up using sparse-representations. In Curves and Surfaces, pages 711–730. 2012. 5, 6 [38] K. Zhang, X. Gao, D. Tao, and X. Li. Single image super-resolution with non-local means and steering kernel regression. IEEE TIP, 21(11):4544–4556, 2012. 6 A Deep Visual Correspondence Embedding Model for Stereo Matching Costs Zhuoyuan Chen, Xun Sun, Liang Wang, Baidu Research – Institute of Deep Learning {chenzhuoyuan,sunxun,wangliang18}@baidu.com Yinan Yu, Chang Huang Horizon Robotics {yinan.yu,chang.huang}@horizon-robotics.com ∗ Abstract This paper presents a data-driven matching cost for stereo matching. A novel deep visual correspondence embedding model is trained via Convolutional Neural Network on a large set of stereo images with ground truth disparities. This deep embedding model leverages appearance data to learn visual similarity relationships between corresponding image patches, and explicitly maps intensity values into an embedding feature space to measure pixel dissimilarities. Experimental results on KITTI and Middlebury data sets demonstrate the effectiveness of our model. First, we prove that the new measure of pixel dissimilarity outperforms traditional matching costs. Furthermore, when integrated with a global stereo framework, our method ranks top 3 among all two-frame algorithms on the KITTI benchmark. Finally, cross-validation results show that our model is able to make correct predictions for unseen data which are outside of its labeled training set. (a) Reference image of a stereo pair (b) Winner-takes-all result (c) Result from an MRF-based stereo framework 1. Introduction Stereo matching infers scene geometry by establishing pixel correspondences across multiple images taken from different viewpoints. Scharstein and Szeliski in their seminal taxonomy work [26] argue that stereo algorithms essentially consist of four building blocks: matching cost computation, cost aggregation, disparity computation/optimization and refinement. According to the actual sequence of steps taken, stereo algorithms can be broadly categorized into local and global methods. Local algorithms, where the disparity computation at a pixel depends only on intensity values within a finite support region, usually make implicit ∗ This work was done when the fourth and fifth authors were with Baidu Research– Institute of Deep Learning. (d) color-coded depth Figure 1. A demonstration of our framework: (a) a left image in the KITTI stereo sequences [13]; (b) we use Convolutional Neural Network to extract discriminative features for stereo matching, which is then refined by a global Markov Random Field to obtain the final result (c). Throughout our paper, we color-code all disparity maps as in (d): blue indicates far regions while white indicates near ones. smoothness assumptions by aggregating pixel-based matching costs. In contrast, global algorithms typically skip the cost aggregation step by making explicit smoothness 1972 assumptions and seek an optimal disparity assignment by solving an MRF (Markov Random Field) based energy optimization problem. Matching cost computation, as a fundamental step shared by both local and global stereo algorithms, plays an important role in establishing visual correspondences. Typically, the reconstruction accuracy of a stereo method largely depends on the dissimilarity measurement of image patches. However, the visual correspondence search problem is difficult due to matching ambiguities, which generally results from sensor noise, image sampling, lighting variations, textureless or repetitive regions, occlusions, etc. In this paper, we focus on the matching cost computation step and present a data-driven approach to address ambiguities. Inspired by the recent advances in deep learning, a new deep visual correspondence embedding model is trained via Convolutional Neural Network on a large set of stereo images with ground truth disparities. This deep embedding model leverages appearance data to learn visual dissimilarity between image patches, by explicitly mapping raw intensity into a rich embedding space. Quantitative evaluations with ground truth data demonstrate the effectiveness of our learned deep embedding model for dissimilarity computation. It is first proved that the proposed measure of pixel dissimilarity outperforms some widely used matching costs such as sampling-insensitive absolute differences [1], absolute differences of gradient, normalized cross-correlation and census transform [41] for local window-based matching. Furthermore, we incorporate our learning-based matching costs with a semi-global method (SGM) [16] and show that our model matches state-of-theart performance on the KITTI stereo data set [13] for accuracy while being superior to other top performers for efficiency and simplicity. Lastly, our deep embedding model is evaluated on the Middlebury benchmark [25] to quantitatively assess its capability of generalization. Crossvalidation results show that our model is able to make correct predictions across stereo images outside of its labeled training set. 1.1. Previous Work Stereo has been a highly active research topic of computer vision for decades and this paper owes a lot to a sizable body of literature on stereo matching, more than we hope to account for here. For the scope of this paper, our focus is on the matching costs computation step and we refer interested readers to [18, 34, 26] for more detailed descriptions and evaluations of different components of stereo algorithms. Common pixel-based matching costs include absolute differences (AD), squared differences (SD) and samplinginsensitive differences (BT) [1]. In practice, truncated versions of these methods are usually adopted because of the robustness brought about by the cost aggregation and global optimization steps. Common window-based matching costs include normalized cross-correlation (NCC), sum of absolute or squared differences (SAD/SSD), gradientbased measures, and non-parametric measures such as rank and census transforms [41]. Methods such as BT, SAD and SSD are built strictly on the brightness constancy assumption while gradient-based and non-parametric measures are more robust to radiometric changes (e.g., due to gain and exposure differences) or non-Lambertian surfaces at the cost of lower discriminative power. Recently, selfadapting dissimilarity measures that combine SAD, Census and gradient-based measures are employed by some stateof-the-art stereo algorithms [19, 4, 39]. CNN (Convolutional Neural Networks) dates back decades in the machine learning community [21] and developed rapidly in recent years, thanks to the advances in learning models [23], larger data sets, and parallel computing devices, as well as the access to many online resources [6, 27]. CNN makes breakthroughs in various computer vision tasks such as image classification [20, 32], object detection [14, 10], face recognition [33, 30], and image parsing [11]. Besides its success in high-level vision applications, deep learning also shows its capability in solving certain low-level vision tasks such as super-resolution [7], denoising [2], and single-view depth estimation [8]. Our work is closely related to a recent paper [35], in which CNN is leveraged to compute stereo matching costs. In comparison to [35], our deep embedding model differs from [35] in two main aspects: 1) Given the feature vectors (corresponding to the left right patches in a stereo pair) output by CNN, we directly compute their similarity in the Euclidean space by a dot product. In contrast, the architecture in [35] is more complicated, in that feature vectors requires further fully-connected DNN (Deep Neural Network) to obtain the final similarity score in a non-Euclidean space. The architecture of our CNN leads to two orders of magnitude (100×) speed-up in computing a dense disparity map compared to [35] with little sacrifice in reconstruction accuracy. 2) Our embedding model is learned from a multiscale ensemble framework, which automatically fuses features vectors learned at different scale-space. Deep learning is also applied in [12, 22] for feature matching. However, [12] extracts features at sparse locations while [22] targets at matching semantically similar regions. Our model is also related to the Deep Flow model [37], where convolutional matching is applied to estimate optical flow. The difference is that [37] applies bottom-up deep dynamic programming on hand-crafted HOG features, while our model is data-driven and learned. 2973 Le#$ image$ patch$ L1$ L2$ L3$ L4$
core1 Share Share Share L1$ L2$ L3$ L4$ L1$ L2$ L3$ L4$
core2 L1$ Share Share Share Right$ image$ patch$ L2$ 1x1x2 voting L3$ L4$ Figure 2. The network architecture of our training model for deep embedding. Features are extracted in a pair of patches at different scales, followed by an inner product to obtain the matching scores. The scores from different scales are then merged for an ensemble. 2. Deep Embedding for Stereo Estimation Given a pair of rectified left/right images {I L , I R }, a typical stereo algorithm begins by computing the matching quality: denoting patches I L (p) and I R (p − d) centered at pL = (x, y) and pR d = (x − d, y), we generate matching score S(p, d) = f (I L (p), I R (p − d)) for each p with different disparity d = (d, 0). The score serves as an important initialization, generally followed by a non-local refinement such as cost aggregation or filtering [24, 42]. In this section, we focus on our data-driven model to learn a good matching quality measure. 2.1. Multi-scale Deep Embedding Model Consider a pair of patches I L (p), I R (p − d) of size 13 × 13, we propose to learn an embedding model to extract features f (I), such that the inner-product S =< f (I L (p)), f (I R (p−d)) > tends to be large in case of positive matching and small for negative ones. This differs from the binary classification model in [35]. Multi-Scale Embedding: we apply an ensemble model [11, 5, 8] in our deep embedding, which largely improves the matching quality. Denoting I↓L (p), I↓R (p−d) as patches at the coarse scale, we have: producing blurred boundaries; small patches have merits in motion details, but are very noisy. Accordingly, we propose a weighted ensemble of two scales to combine the best of two worlds. Our embedding framework is shown in Figure 2. The inputs are two pairs of 13 × 13 × 1 image patches, centered at p = (x, y), p − d = (x − d, y) with two different scales. The blue-colored dash box indicates the original resolution while the red is a ×2 down-sampling. We apply a 4-layer CNN model to extract features f (I), followed by an inner-product layer to calculate the matching score < f (I L ), f (I R ) > and an ensemble voting. Layer L1 and L2 contain c1 = c2 = 32 kernels of size 3 × 3; Layer L3 and L4 contain kernels c3 = c4 = 200 of size 5 × 5. At both scales, the weights in all layers L1...L4 for left and right patches are tied. Then, features f4 (I) ∈ R200 undergo inner-product operations, outputting two scalars S1 =< f4 (I L ), f4 (I R ) > and S2 =< f4 (I↓L ), f4 (I↓R ) > as matching scores. Finally, S1 and S2 are merged by a 1 × 1 × 2 convolutional layer for a weighted ensemble. In our training process, we apply a deep regression model and minimize the Euclidean cost: E(w) = ||S(p, d) − label(p, d)||2 S(p, d) = w1 < f (I L (p), f (I R (p − d)) > + w2 < f (I↓L (p), f (I↓R (p − d) > As we know, the choice of patch scale and size is very tricky: large patches with richer information are less ambiguous, but more risky of containing multiple objects and (1) where label(p, d) = {0, 1} indicates whether pL = (x, y) corresponds to p − d = (x − d, y) in the right image. Rectified linear units [23] follow each layer, but we do not use pooling to preserve spatial-variance. 3974 First of all, a cost volume is initialized by negating our deep embedding scores as C(p, d) = −S(p, pd ). Secondly, the initial costs are then fed to the semi-global matcher (SGM) [16, 39] to compute a raw disparity map. Thirdly, after removing unreliable matches via a left-right check, the final disparity map is obtained by propagating reliable disparities to non-reliable areas [29]. In the following, we will briefly summarize the key building blocks in our framework. Formulating the stereo problem as an MRF model, the SGM aims to find a disparity assignment D that minimizes: Figure 3. The deployed network architecture of our testing model for deep embedding. Features are extracted in two images only once respectively. Then, the sliding-window style inner product can be grouped for matrix operation. E(D) = X (C(p, d) + + X p X P1 [|d − Dq | = 1] q∈N (p) (2) P2 [|d − Dq | > 1]) q∈N (p) 2.2. Efficient Embedding for Testing With deep embedding, we achieve 100× speedup at test time compared to MC-CNN [35]. We attribute this to the largely shared computation in the forward pass as well as grouped matrix operation. The insight is shown in Figure 3, we extract features f4 (I) in each image only once with a fully convolutional neural network, and then the matching score S(p, d) is computed with all potential offsets d by the sliding-window style inner product. In comparison, in [35] the fullyconnected layers require recomputation every time when d is varied. Moreover, multiple inner product operations can be grouped together as a matrix operation for further acceleration. That is, we multiply f (I L (p)) with the matrix F (I R ), whose columns contain features f (I R (p − d)) with different d. This matrix multiplication is highly parallel in nature. 2.3. Training Details A training example comprises two pairs of patches {I L (p), I R (p − d)}, {I↓L (p), I↓R (p − d)}. We sample positive and negative examples at ratio 1 : 1 at each location where the disparity d is known. A negative example is obtained by shifting the right patch to pd = (x − d + oneg , y), where oneg ∈ {−Nhi , ..., −Nlo , Nlo , ..., Nhi } is a random corrupting offset. Similarly, positive examples are pd = (x − d + opos , y), with opos ∈ {−Phi , Phi }. In practice, we find that a warm start with large Nlo , Nhi makes the training converges faster. We gradually decrease Nlo , Nhi to pursue a hard negative mining. 3. Stereo Framework To calculate the final disparity map using our proposed matching costs, we adopt an MRF-based stereo framework. The constant parameters P1 and P2 penalize disparity discontinuity and P1 < P2 . While an exact solution to equation (2) is NP-hard, semi-global matcher obtains an approximation by performing multiple dynamic programming (DP)-style 1D cost updates along multiple directions as Lr (p, d) =C(p, d) + min(Lr (p − r, d), Lr (p − r, d ± 1) + P1 , (3) min(Lr (p − r, i) + P2 )) i where Lr (p, d) is the cost along a direction r. After averaging the costs along 16 directions, the initial disparity map is computed for left and right views by picking the disparity hypothesis with the minimal cost. To efficiently remove the remaining outliers in the initial disparity map, we adopt a propagation scheme [29, 40]. We start by performing a left-right consistency check to roughly divide all pixels into stable and unstable sets. Assuming that a stable pixel p should satisfy DLef t (p) = DRight (p − d), we carry out a propagation on the cost volume as: C(p, d)pro = ⇢ Lef t |d − DRaw (p)|2 0 p is stable, else. (4) Here for stable pixels, we penalize disparity deviation from its initial disparity values. In contrast, for unstable pixels, the costs are set to zero for all disparity hypotheses. Thus, an edge-aware filter applied on this cost volume leads to reliable disparity propagation in the cost domain. We choose the O(1) geodesic filter derived from a tree structure [29] due to its effectiveness and efficiency. After filtering the propagated cost volume C(p, d)pro , we again choose the disparity assignment using winner-takes-all. Finally, we employ a 3 × 3 median filter to remove remaining isolated noises. 4975 (a) Image Sequence (b) Low-resolution d↓ , accuracy 94.2% (c) High-resolution d, accuracy 89.6% (d) Ensemble d∗ , accuracy 95.8% (e) Refined result of d∗ (Section 3), accuracy 98.5% (f) (g) (h) (i) Figure 4. An example of how the proposed multi-scale ensemble works. (a) Frame 10 in KITTI [13]; (b,c) winner-takes-all results of embedding at the coarse and fine scales, respectively; (d) ensemble of (b,c) gives smooth results while preserving details at the same time; (f,g,h,i) are close-up views of (b,c,d,e), respectively. 4. Experimental Results In this section, we evaluate our algorithm extensively by comparing its performance with other state-of-the-art methods. Our matching costs computation is implemented in CUDA. And the subsequent stereo framework is implemented on a PC equipped with a 3.0 GHz Intel i5 CPU. In our stereo framework, we fix the parameter settings throughout our experiments: {P1 , P2 , σs , σr } = {5, 80, 20, 10.5}. Here σs and σr are the spatial and range kernel parameters used in geodesic filters, respectively. We use off-the-shelf resource Caffe [6] to train our CNN with a Nvidia GeForce GTX Titan. We optimize the Euclidean loss by a stochastic gradient descent with momentum and weight decay [31]. Specially, we preprocess each image by subtracting the mean and dividing by the standard deviation of its pixel intensity values. It takes about 3 hours to train the CNN with KITTI training set. 4.1. Matching Cost Evaluation We start by comparing our learning-based matching costs against traditional pixel and window-based matching costs, including BT (Bircheld and Tomasi) [1], census transform [41], AD+Gradient (combination of absolute differences and gradient-based measures), Census+Gradient (combination of census transform and gradient-based measures) [39], and NCC. Among these selected counterparts, BT and AD+Gradient are popular choices by many competitive stereo algorithms on the Middlebury evaluation system [19, 24]; census transform is reported as the overall best matching cost function in Hirschmuller and Scharstein’s survey [18]; Census+Gradient is used by one of the top performers on the outdoor KITTI data sets [39]; NCC is a classic patch-based measure that is statistically optimal for compensating Gaussian noise [18]. For AD+Gradient and Census+Gradient, we follow the parameter settings as reported in the original literature [19, 39], census transform is computed with a 5×5 local support region. Similar to [18], we use a square window to aggregate the matching costs and calculate the per-pixel disparity value via winner-takes-all. The window size is chosen to be 13 × 13 because our deep embedding model is learned on the same size. NCC does not go through cost aggregation with the patch width set to 13. Note that no disparity refinement step is applied in this experiment because we want the raw results to provide a more direct assessment of different methods. We report performance on the 194 KITTI training images in Table 1. As can be seen, our learned matching costs from the deep embedding model substantially outperform traditional matching costs. Validation of Multi-scale Ensemble: Close scrutiny reveals the complementary power of the embedding results 5976 Figure 5. Some disparity map results using our embedding cost on KITTI benchmark. The left column: grayscale images of the left frames; The right column: our final results. From top to bottom: different frames in the KITTI data set. Methods Multi-scale Single-scale (Coarse) Single-scale (Fine) BT Census AD+Gradient Census+Gradient NCC All err(%) 11.1 14.8 17.4 37.8 25.1 37.4 23.5 21.8 Non-occ err(%) 9.5 12.7 14.6 36.6 23.4 36.1 21.7 19.9 Table 1. Quantitative evaluation of different costs with localwindow matching. We compare errors of winner-takes-all results obtained by classical costs with our deep embedding model (single-scale and the multi-scale ensembles) in all and nonoccluded regions. at different scales (e.g., the 10-th frame in KITTI training set). As shown in Figure 4, the embedding at lowresolution tends to generate more smooth results, but is at risk of losing motion details. Thin structures like the tree marked in box in (a) are missing in (b,f), where the lost mo- tion proposal can hardly be recovered by post-processing. Meanwhile, noises in high-resolution results (c) can be suppressed by accounting for larger contexts. The accuracy of ensemble (95.8%) is higher than any single scale and a postprocessing can produce satisfactory stereo results. Quantitatively, the average accuracy of winner-takes-all results produced by the fine-scale, coarse-scale and multiscale ensemble are 85.4%, 87.3% and 90.5% on the KITTI train set, respectively. 4.2. Stereo Pipeline Evaluation In this section, we incorporate our matching costs with the stereo framework introduced in section 3. The quantitative evaluation results on KITTI test set are shown in table 2, where we compare our model with state-of-the-art methods with only two stereo images for fairness (algorithms that leverage multi-view or temporal/scene flow cues are beyond the scope of this paper). Our method ranks the 3rd among these algorithms in terms of accuracy as of September 2015, and is over an order of magnitude faster than [15] and [35]. Note that replacing our deep embedding costs 6977 Method Displets [15] MC-CNN [35] Ours Census+Gradient CoR[3] SPS-St[39] DDS-SS[36] PCBP[38] CoR-Conf[3] AARBM[9] wSGM [28] Out-Noc 2.47 % 2.61 % 3.10 % 3.91 % 3.30 % 3.39% 3.83 % 4.04 % 4.49 % 4.86 % 4.97 % Out-All 3.27 % 3.84 % 4.24 % 5.10 % 4.10 % 4.41% 4.59 % 5.37 % 5.26 % 5.94 % 6.18 % Avg-Noc 0.7 px 0.8 px 0.9 px 0.9 px 0.8 px 0.9 px 0.9 px 0.9 px 1.0 px 1.0 px 1.3 px Avg-All 0.9 px 1.0 px 1.1 px 1.0 px 0.9 px 1.0 px 1.0 px 1.1 px 1.2 px 1.2 px 1.6 px Runtime 265s 100s 3.0s 2.5s 6s 2s 1 min 5 min 6s 0.25 s 6s Environment 8+ cores @ 3.0 Ghz (Matlab + C/C++) Nvidia GTX Titan (CUDA, Lua/Torch7) Nvidia GTX Titan (CUDA, Caffe) single core @ 3.0Ghz GPU 6 cores @ 3.3 Ghz (Matlab + C/C++) 1 core @3.5 Ghz (C/C++) 1 core @ 2.5 Ghz (Matlab + C/C++) 4 cores @ 2.5 Ghz (Matlab + C/C++) 6 cores @ 3.3 Ghz (Matlab + C/C++) 1 core @ 3.0 Ghz (C/C++) 1 core @ 3.5 Ghz (C/C++) Table 2. Qualitative evaluation of our pipeline on KITTI benchmark with the error threshold set as 3 pixels. The method ”Census+Gradient” uses the same stereo framework, but with Census+Gradient cost function rather than our proposed learning-based matching costs. In terms of runtime, we would like to emphasize that the numbers given in this table are total running time of the whole stereo framework. As reported in the paper [35], the MC-CNN algorithm takes about 95 seconds to compute their learning-based matching costs and 5 seconds for semi-global matching and disparity post-processing steps. In contrast, the running time for our matching cost computation is about 1.0 second, which is around 100 times faster than [35]. with Census+Gradient [39] produces inferior results. A few disparity maps generated by our method are presented in Figure 5, from which we can see that our algorithm produces piecewise smooth and visually plausible results. Our method not only preserves geometry details near depth discontinuities, but also performs well on challenging regions such as textureless and shadow areas. Regarding the running time, the CNN step takes about 1.0s on average, 734ms at the original resolution and 267ms at the coarse resolution. This is about 100× speedup compared to MC-CNN [35]. The acceleration factor mainly results from fewer model parameters (116,000 versus 600,000 in [35]), fully-convolutional architecture and grouped matrix operation. The not fully optimized MRF stereo implementation takes about 2s on CPU. In total, our approach takes about 3s to process an image at the resolution of 1242 × 376 with a large disparity searching range from 0 to 255. 4.3. Cross-Validation It is of interest to evaluate how our model performs on different datasets. Besides KITTI (outdoor street view), we further test our deep embedding model on indoor scenarios without any fine-tuning. We choose stereo sequences from Middlebury benchmarks [25, 17], containing a total number of 27 high resolution image pairs. Quantitatively, we compare our algorithm with traditional costs such as AD, census transform and NCC. We evaluate the performance with varying error thresholds τ in Figure 6: similar to the setup in section 4.1, we carry out a box aggregation on raw costs and obtain final disparities with a winner-takes-all treatment. It is clear that our embedding model consistently outperforms other methods. Especially, with τ = 3, our method achieves an average er- Figure 6. The error curve on Middlebury benchmark with varying error thresholds. ror rate of 13.6%, which is much lower than SAD (33.8%), census transform (22.2%), and NCC (21.9%). This demonstrates that our deep embedding model generalizes well on scenes that are sufficiently different from the training data. We provide more qualitative results in Figure 7. 5. Conclusion and Future Work In this paper, we introduce a novel data-driven patch dissimilarity measure for visual correspondence. We learn a deep embedding model to extract discriminative features from patches in stereo pairs. Our model has fewer parameters and shallower network structures therefore is much more efficient than a previous learning-based model [35]. Deep embedding produces high-quality initialization, which can be refined with an MRF-based stereo algorithm to obtain the state-of-the-art dense disparity estimates. Our results on KITTI [13] and Middlebury [26] benchmarks 7978 Figure 7. Examples of winner-takes-all results using our deep embedding cost on Middlebury data. From left to right: the Cloth, Dolls, Books and Moebius Sequences in Middlebury benchmark; From top to bottom: the left images, winner-takes-all results of our deep embedding and ground truth. suggest that deep embedding is robust across stereo images taken from different environment. In the future, training with larger data sets might have potential to further improve the matching accuracy. We will also accelerate the algorithm for real-time navigation and extend our model to solve other low-level vision problems such as optical flow. 6. Acknowledgements We would like to thank Haoyuan Gao and Fei Qing for their efforts in data collection. We would also like to thank Kai Yu and Carl Case for discussion and helpful advice. References [1] S. Birchfield and C. Tomasi. A pixel dissimilarity measure that is insensitive to image sampling. PAMI, 20:401–406, 1998. [2] H. C. Burger, C. J. Schuler, and S. Harmeling. Image denoising: Can plain neural networks compete with bm3d? CVPR, pages 2392–2399, 2012. [3] A. Chakrabarti, Y. Xiong, S. J. Gortler, and T. Zickler. Lowlevel vision by consensus in a spatial hierarchy of regions. CVPR, 2015. [4] Z. Chen, H. Jin, Z. Lin, S. Cohen, and Y. Wu. Large displacement optical flow from nearest neighbor fields. CVPR, 2013. [5] D. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. CVPR, pages 3642–3649, 2012. [6] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. ICML, 2014. [7] C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. ECCV, 2014. [8] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. NIPS, pages 2366–2374, 2014. [9] N. Einecke and J. Eggert. Block-matching stereo with relaxed fronto-parallel assumption. In In Intelligent Vehicles Symposium Proceedings, 2014 IEEE, pages 700–705, 2014. [10] D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov. Scalable object detection using deep neural networks. CVPR, 2014. [11] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. PAMI, 35(8):1915– 1929, 2013. [12] P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor matching with convolutional neural networks: a comparison to sift. Technical Report 1405.5769, arXiv(1405.5769), 2014. 8979 [13] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. [14] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR, 2014. [15] F. Güney and A. Geiger. Displets: Resolving stereo ambiguities using object knowledge. In CVPR, 2015. [16] H. Hirschmüller. Stereo processing by semiglobal matching and mutual information. PAMI, 30(2):328–341, 2008. [17] H. Hirschmuller and D. Scharstein. Evaluation of cost functions for stereo matching. CVPR, 2007. [18] H. Hirschmuller and D. Scharstein. Evaluation of stereo matching costs on images with radiometric differences. PAMI, 31:1582–1599, 2009. [19] A. Klaus, M. Sormann, and K. Karner. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. ICPR, 2006. [20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NIPS, 2012. [21] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [22] J. Long, N. Zhang, and T. Darrell. Do convnets learn correspondence? NIPS, 2014. [23] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. ICML, 2010. [24] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR, 2011. [25] D. Scharstein and C. Pal. Learning conditional random fields for stereo. CVPR, 2007. [26] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV, 47(1-3):7–42, 2002. [27] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013. [28] R. Spangenberg, T. Langner, and R. Rojas. Weighted semiglobal matching and center-symmetric census transform for robust driver assistance. In In Computer Analysis of Images and Patterns, pages 34–41, 2013. [29] X. Sun, X. Mei, S. Jiao, M. Zhou, Z. Liu, and H. Wang. Realtime local stereo via edge-aware disparity propagation. PRL, 49:201–206, 2014. [30] Y. Sun, X. Wang, and X. Tang. Deep learning face representation from predicting 10,000 classes. CVPR, 2014. [31] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. ICML, 2013. [32] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv:1409.4842, 2014. [33] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. CVPR, 2014. [34] F. Tombari, S. Mattoccia, L. D. Stefano, and E. Addimanda. Classification and evaluation of cost aggregation methods for stereo correspondence. CVPR, 2008. [35] J. Z̆bontar and Y. LeCun. Computing the stereo matching cost with a convolutional neural network, 2014. [36] D. Wei, C. Liu, and W. T. Freeman. A data-driven regularization model for stereo and flow. In 3D Vision (3DV), 2014 2nd International Conference on, 1:277–284, 2014. [37] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. Deepflow: Large displacement optical flow with deep matching. ICCV, pages 1385–1392, 2013. [38] K. Yamaguchi, T. Hazan, D. McAllester, and R. Urtasun. Continuous markov random fields for robust stereo estimation. In ECCV, pages 45–58, 2012. [39] K. Yamaguchi, D. McAllester, and R. Urtasun. Efficient joint segmentation, occlusion labeling, stereo and flow estimation. ECCV, 2013. [40] Q. Yang. A non-local cost aggregation method for stereo matching. In CVPR, 2012. [41] R. Zabih and J. Woodfill. Non-parametric local transforms for computing visual correspondence. ECCV, 1994. [42] K. Zhang, J. Lu, and G. Lafruit. Cross-based local stereo matching using orthogonal integral images. IEEE Transactions on Circuits and Systems for Video Technology, 19(7):1073–1079, 2009. 9980 Conditional Random Fields as Recurrent Neural Networks Shuai Zheng∗1 , Sadeep Jayasumana*1 , Bernardino Romera-Paredes1 , Vibhav Vineet†1,2 , Zhizhong Su3 , Dalong Du3 , Chang Huang3 , and Philip H. S. Torr1 1 University of Oxford 2 Stanford University Abstract Baidu Institute of Deep Learning lenge in pixel-level labelling problems. Work on this topic includes: TextonBoost [52], TextonForest [51], and Random Forest-based classifiers [50]. Recently, supervised deep learning approaches such as large-scale deep Convolutional Neural Networks (CNNs) have been immensely successful in many high-level computer vision tasks such as image recognition [31] and object detection [20]. This motivates exploring the use of CNNs for pixel-level labelling problems. The key insight is to learn a strong feature representation end-to-end for the pixel-level labelling task instead of hand-crafting features with heuristic parameter tuning. In fact, a number of recent approaches including the particularly interesting works FCN [37] and DeepLab [10] have shown a significant accuracy boost by adapting stateof-the-art CNN based image classifiers to the semantic segmentation problem. Pixel-level labelling tasks, such as semantic segmentation, play a central role in image understanding. Recent approaches have attempted to harness the capabilities of deep learning techniques for image recognition to tackle pixellevel labelling tasks. One central issue in this methodology is the limited capacity of deep learning techniques to delineate visual objects. To solve this problem, we introduce a new form of convolutional neural network that combines the strengths of Convolutional Neural Networks (CNNs) and Conditional Random Fields (CRFs)-based probabilistic graphical modelling. To this end, we formulate mean-field approximate inference for the Conditional Random Fields with Gaussian pairwise potentials as Recurrent Neural Networks. This network, called CRF-RNN, is then plugged in as a part of a CNN to obtain a deep network that has desirable properties of both CNNs and CRFs. Importantly, our system fully integrates CRF modelling with CNNs, making it possible to train the whole deep network end-to-end with the usual back-propagation algorithm, avoiding offline post-processing methods for object delineation. We apply the proposed method to the problem of semantic image segmentation, obtaining top results on the challenging Pascal VOC 2012 segmentation benchmark. However, there are significant challenges in adapting CNNs designed for high level computer vision tasks such as object recognition to pixel-level labelling tasks. Firstly, traditional CNNs have convolutional filters with large receptive fields and hence produce coarse outputs when restructured to produce pixel-level labels [37]. Presence of maxpooling layers in CNNs further reduces the chance of getting a fine segmentation output [10]. This, for instance, can result in non-sharp boundaries and blob-like shapes in semantic segmentation tasks. Secondly, CNNs lack smoothness constraints that encourage label agreement between similar pixels, and spatial and appearance consistency of the labelling output. Lack of such smoothness constraints can result in poor object delineation and small spurious regions in the segmentation output [59, 58, 32, 39]. 1. Introduction Low-level computer vision problems such as semantic image segmentation or depth estimation often involve assigning a label to each pixel in an image. While the feature representation used to classify individual pixels plays an important role in this task, it is similarly important to consider factors such as image edges, appearance consistency and spatial consistency while assigning labels in order to obtain accurate and precise results. Designing a strong feature representation is a key chal- On a separate track to the progress of deep learning techniques, probabilistic graphical models have been developed as effective methods to enhance the accuracy of pixellevel labelling tasks. In particular, Markov Random Fields (MRFs) and its variant Conditional Random Fields (CRFs) have observed widespread success in this area [32, 29] and have become one of the most successful graphical models used in computer vision. The key idea of CRF inference for semantic labelling is to formulate the label assignment ∗ Authors † Work 3 contributed equally. conducted while authors at the University of Oxford. 1 problem as a probabilistic inference problem that incorporates assumptions such as the label agreement between similar pixels. CRF inference is able to refine weak and coarse pixel-level label predictions to produce sharp boundaries and fine-grained segmentations. Therefore, intuitively, CRFs can be used to overcome the drawbacks in utilizing CNNs for pixel-level labelling tasks. One way to utilize CRFs to improve the semantic labelling results produced by a CNN is to apply CRF inference as a post-processing step disconnected from the training of the CNN [10]. Arguably, this does not fully harness the strength of CRFs since it is not integrated with the deep network. In this setup, the deep network is unaware of the CRF during the training phase. In this paper, we propose an end-to-end deep learning solution for the pixel-level semantic image segmentation problem. Our formulation combines the strengths of both CNNs and CRF based graphical models in one unified framework. More specifically, we formulate mean-field approximate inference for the dense CRF with Gaussian pairwise potentials as a Recurrent Neural Network (RNN) which can refine coarse outputs from a traditional CNN in the forward pass, while passing error differentials back to the CNN during training. Importantly, with our formulation, the whole deep network, which comprises a traditional CNN and an RNN for CRF inference, can be trained endto-end utilizing the usual back-propagation algorithm. Arguably, when properly trained, the proposed network should outperform a system where CRF inference is applied as a post-processing method on independent pixel-level predictions produced by a pre-trained CNN. Our experimental evaluation confirms that this indeed is the case. We evaluate the performance of our network on the popular Pascal VOC 2012 benchmark, achieving a new state-of-the-art accuracy of 74.7%. super-pixels) may lead to poor predictions, no matter how good the feature extraction process is. Pinheiro and Collobert [46] employed an RNN to model the spatial dependencies during scene parsing. In contrast to their approach, we show that a typical graphical model such as a CRF can be formulated as an RNN to form a part of a deep network, to perform end-to-end training combined with a CNN. The second strategy is to directly learn a nonlinear model from the images to the label map. This, for example, was shown in [17], where the authors replaced the last fully connected layers of a CNN by convolutional layers to keep spatial information. An important contribution in this direction is [37], where Long et al. used the concept of fully convolutional networks, and the notion that top layers obtain meaningful features for object recognition whereas low layers keep information about the structure of the image, such as edges. In their work, connections from early layers to later layers were used to combine these cues. Bell et al. [5] and Chen et al. [10, 41] used a CRF to refine segmentation results obtained from a CNN. Bell et al. focused on material recognition and segmentation, whereas Chen et al. reported very significant improvements on semantic image segmentation. In contrast to these works, which employed CRF inference as a standalone post-processing step disconnected from the CNN training, our approach is an end-to-end trainable network that jointly learns the parameters of the CNN and the CRF in one unified deep network. Works that use neural networks to predict structured output are found in different domains. For example, Do et al. [14] proposed an approach to combine deep neural networks and Markov networks for sequence labeling tasks. Jain et al. [26] has shown Convolutional Neural Networks can perform well like MRFs/CRFs approaches in image restoration application. Another domain which benefits from the combination of CNNs and structured loss is handwriting recognition. In natural language processing, Yao et al. [60] shows that the performance of an RNN-based words tagger can be significantly improved by incorporating elements of the CRF model. In [6], the authors combined a CNN with Hidden Markov Models for that purpose, whereas more recently, Peng et al. [45] used a modified version of CRFs. Related to this line of works, in [25] a joint CNN and CRF model was used for text recognition on natural images. Tompson et al. [57] showed the use of joint training of a CNN and an MRF for human pose estimation, while Chen et al. [11] focused on the image classification problem with a similar approach. Another prominent work is [21], in which the authors express deformable part models, a kind of MRF, as a layer in a neural network. In our approach, we cast a different graphical model as a neural network layer. A number of approaches have been proposed for automatic learning of graphical model parameters and joint 2. Related Work In this section we review approaches that make use of deep learning and CNNs for low-level computer vision tasks, with a focus on semantic image segmentation. A wide variety of approaches have been proposed to tackle the semantic image segmentation task using deep learning. These approaches can be categorized into two main strategies. The first strategy is based on utilizing separate mechanisms for feature extraction, and image segmentation exploiting the edges of the image [2, 38]. One representative instance of this scheme is the application of a CNN for the extraction of meaningful features, and using superpixels to account for the structural pattern of the image. Two representative examples are [19, 38], where the authors first obtained superpixels from the image and then used a feature extraction process on each of them. The main disadvantage of this strategy is that errors in the initial proposals (e.g: 2 training of classifiers and graphical models. Barbu et al. [4] proposed a joint training of a MRF/CRF model together with an inference algorithm in their Active Random Field approach. Domke [15] advocated back-propagation based parameter optimization in graphical models when approximate inference methods such as mean-field and belief propagation are used. This idea was utilized in [28], where a binary dense CRF was used for human pose estimation. Similarly, Ross et al. [47] and Stoyanov et al. [54] showed how back-propagation through belief propagation can be used to optimize model parameters. Ross et al. [21], in particular proposes an approach based on learning messages. Many of these ideas can be traced back to [55], which proposes unrolling message passing algorithms as simpler operations that could be performed within a CNN. In a different setup, Krähenbühl and Koltun [30] demonstrated automatic parameter tuning of dense CRF when a modified mean-field algorithm is used for inference. An alternative inference approach for dense CRF, not based on mean-field, is proposed in [61]. In contrast to the works described above, our approach shows that it is possible to formulate dense CRF as an RNN so that one can form an end-to-end trainable system for semantic image segmentation which combines the strengths of deep learning and graphical modelling. After our initial publication of the technical report of this work on arXiv.org, a number of independent works [49, 35] appeared on arXiv.org presenting similar joint training approaches for semantic image segmentation. Figure 1. A mean-field iteration as a CNN. A single iteration of the mean-field algorithm can be modelled as a stack of common CNN layers. energy of a label assignment x is given by: X X E(x) = ψu (xi ) + ψp (xi , xj ), i (1) i<j where the unary energy components ψu (xi ) measure the inverse likelihood (and therefore, the cost) of the pixel i taking the label xi , and pairwise energy components ψp (xi , xj ) measure the cost of assigning labels xi , xj to pixels i, j simultaneously. In our model, unary energies are obtained from a CNN, which, roughly speaking, predicts labels for pixels without considering the smoothness and the consistency of the label assignments. The pairwise energies provide an image data-dependent smoothing term that encourages assigning similar labels to pixels with similar properties. As was done in [29], we model pairwise potentials as weighted Gaussians: ψp (xi , xj ) = µ(xi , xj ) M X (m) w(m) kG (fi , fj ), (2) m=1 3. Conditional Random Fields (m) where each kG for m = 1, . . . , M , is a Gaussian kernel applied on feature vectors. The feature vector of pixel i, denoted by fi , is derived from image features such as spatial location and RGB values [29]. We use the same features as in[29]. The function µ(., .), called the label compatibility function, captures the compatibility between different pairs of labels as the name implies. Minimizing the above CRF energy E(x) yields the most probable label assignment x for the given image. Since this exact minimization is intractable, a mean-field approximation to the CRF distribution is used for approximate maximum posterior marginal inference. It consists in approximating the CRF distribution P (X) by a simpler distribution Q(X), which can be written as the product of independent Q marginal distributions, i.e., Q(X) = i Qi (Xi ). The steps of the iterative algorithm for approximate mean-field inference and its reformulation as an RNN are discussed next. In this section we provide a brief overview of Conditional Random Fields (CRF) for pixel-wise labelling and introduce the notation used in the paper. A CRF, used in the context of pixel-wise label prediction, models pixel labels as random variables that form a Markov Random Field (MRF) when conditioned upon a global observation. The global observation is usually taken to be the image. Let Xi be the random variable associated to pixel i, which represents the label assigned to the pixel i and can take any value from a pre-defined set of labels L = {l1 , l2 , . . . , lL }. Let X be the vector formed by the random variables X1 , X2 , . . . , XN , where N is the number of pixels in the image. Given a graph G = (V, E), where V = {X1 , X2 , . . . , XN }, and a global observation (image) I, the pair (I, X) can be modelled as a CRF characterized by a Gibbs distribution of the form P (X = x|I) = 1 Z(I) exp(−E(x|I)). Here E(x) is called the energy of the configuration x ∈ LN and Z(I) is the partition function [33]. From now on, we drop the conditioning on I in the notation for convenience. In the fully connected pairwise CRF model of [29], the 4. A Mean-field Iteration as a Stack of CNN Layers A key contribution of this paper is to show that the meanfield CRF inference can be reformulated as a Recurrent 3 4.1. Initialization Algorithm 1 Mean-field in dense CRFs [29], broken down to common CNN operations. In the initialization step of the algorithm, P the operation Qi (l) ← Z1i exp (Ui (l)), where Zi = l exp(Ui (l)), is performed. Note that this is equivalent to applying a softmax function over the unary potentials U across all the labels at each pixel. The softmax function has been extensively used in CNN architectures before and is therefore well known in the deep learning community. This operation does not include any parameters and the error differentials received at the output of the step during back-propagation could be passed down to the unary potential inputs after performing usual backward pass calculations of the softmax transformation. . Initialization Qi (l) ← Z1i exp (Ui (l)) for all i while not converged do P (m) Q̃i (l) ← j6=i k(m) (fi , fj )Qj (l) for all m . Message Passing P (m) Q̌i (l) ← m w(m) Q̃i (l) . Weighting Filter Outputs P Q̂i (l) ← l0 ∈L µ(l, l0 )Q̌i (l0 ) . Compatibility Transform Q̆i (l) ← Ui (l) − Q̂i (l) . Adding Unary Potentials   Qi ← Z1i exp Q̆i (l) . Normalizing end while 4.2. Message Passing In the dense CRF formulation, message passing is implemented by applying M Gaussian filters on Q values. Gaussian filter coefficients are derived based on image features such as the pixel locations and RGB values, which reflect how strongly a pixel is related to other pixels. Since the CRF is potentially fully connected, each filter’s receptive field spans the whole image, making it infeasible to use a brute-force implementation of the filters. Fortunately, several approximation techniques exist to make computation of high dimensional Gaussian filtering significantly faster. Following [29], we use the Permutohedral lattice implementation [1], which can compute the filter response in O(N ) time, where N is the number of pixels of the image [1]. During back-propagation, error derivatives w.r.t. the filter inputs are calculated by sending the error derivatives w.r.t. the filter outputs through the same M Gaussian filters in reverse direction. In terms of permutohedral lattice operations, this can be accomplished by only reversing the order of the separable filters in the blur stage, while building the permutohedral lattice, splatting, and slicing in the same way as in the forward pass. Therefore, back-propagation through this filtering stage can also be performed in O(N ) time. Following [29], we use two Gaussian kernels, a spatial kernel and a bilateral kernel. In this work, for simplicity, we keep the bandwidth values of the filters fixed. It is also possible to use multiple spatial and bilateral kernels with different bandwidth values and learn their optimal linear combination. Neural Network (RNN). To this end, we first consider individual steps of the mean-field algorithm summarized in Algorithm 1 [29], and describe them as CNN layers. Our contribution is based on the observation that filter-based approximate mean-field inference approach for dense CRFs relies on applying Gaussian spatial and bilateral filters on the mean-field approximates in each iteration. Unlike the standard convolutional layer in a CNN, in which filters are fixed after the training stage, we use edge-preserving Gaussian filters [56, 42], coefficients of which depend on the original spatial and appearance information of the image. These filters have the additional advantages of requiring a smaller set of parameters, despite the filter size being potentially as big as the image. While reformulating the steps of the inference algorithm as CNN layers, it is essential to be able to calculate error differentials in each layer w.r.t. its inputs in order to be able to back-propagate the error differentials to previous layers during training. We also discuss how to calculate error differentials with respect to the parameters in each layer, enabling their optimization through the back-propagation algorithm. Therefore, in our formulation, CRF parameters such as the weights of the Gaussian kernels and the label compatibility function can also be optimized automatically during the training of the full network. Once the individual steps of the algorithm are broken down as CNN layers, the full algorithm can then be formulated as an RNN. We explain this in Section 5 after discussing the steps of Algorithm 1 in detail below. In Algorithm 1 and the remainder of this paper, we use Ui (l) to denote the negative of the unary energy introduced in the previous section, i.e., Ui (l) = −ψu (Xi = l). In the conventional CRF setting, this input Ui (l) to the mean-field algorithm is obtained from an independent classifier. 4.3. Weighting Filter Outputs The next step of the mean-field iteration is taking a weighted sum of the M filter outputs from the previous step, for each class label l. When each class label is considered individually, this can be viewed as usual convolution with a 1 × 1 filter with M input channels, and one output channel. Since both inputs and the outputs to this step are known during back-propagation, the error derivative w.r.t. the filter 4 4.6. Normalization weights can be computed, making it possible to automatically learn the filter weights (relative contributions from each Gaussian filter output from the previous stage). Error derivative w.r.t. the inputs can also be computed in the usual manner to pass the error derivatives down to the previous stage. To obtain a higher number of tunable parameters, in contrast to [29], we use independent kernel weights for each class label. The intuition is that the relative importance of the spatial kernel vs the bilateral kernel depends on the visual class. For example, bilateral kernels may have on the one hand a high importance in bicycle detection, because similarity of colours is determinant; on the other hand they may have low importance for TV detection, given that whatever is inside the TV screen may have many different colours. Finally, the normalization step of the iteration can be considered as another softmax operation with no parameters. Differentials at the output of this step can be passed on to the input using the softmax operation’s backward pass. 5. The End-to-end Trainable Network We now describe our end-to-end deep learning system for semantic image segmentation. To pave the way for this, we first explain how repeated mean-field iterations can be organized as an RNN. 5.1. CRF as RNN In the previous section, it was shown that one iteration of the mean-field algorithm can be formulated as a stack of common CNN layers (see Fig. 1). We use the function fθ to denote the transformation done by one mean-field iteration: given an image I, pixel-wise unary potential values U and an estimation of marginal probabilities Qin from the previous iteration, the next estimation of marginal distributions after one mean-field iteration is given by fθ (U, Qin , I). 0 0 The vector θ = {w(m) , µ(l, l )}, m ∈ {1, ..., M }, l, l ∈ {l1 , ..., lL } represents the CRF parameters described in Section 4. Multiple mean-field iterations can be implemented by repeating the above stack of layers in such a way that each iteration takes Q value estimates from the previous iteration and the unary values in their original form. This is equivalent to treating the iterative mean-field inference as a Recurrent Neural Network (RNN) as shown in Fig. 2. Using the notation in the figure, the behaviour of the network is given by the following equations where T is the number of mean-field iterations: 4.4. Compatibility Transform In the compatibility transform step, outputs from the previous step (denoted by Q̌ in Algorithm 1) are shared between the labels to a varied extent, depending on the compatibility between these labels. Compatibility between the two labels l and l0 is parameterized by the label compatibility function µ(l, l0 ). The Potts model, given by µ(l, l0 ) = [l 6= l0 ], where [.] is the Iverson bracket, assigns a fixed penalty if different labels are assigned to pixels with similar properties. A limitation of this model is that it assigns the same penalty for all different pairs of labels. Intuitively, better results can be obtained by taking the compatibility between different label pairs into account and penalizing the assignments accordingly. For example, assigning labels “person” and “bicycle” to nearby pixels should have a lesser penalty than assigning labels “sky” and “bicycle”. Therefore, learning the function µ from data is preferred to fixing it in advance with Potts model. We also relax our compatibility transform model by assuming that µ(l, l0 ) 6= µ(l0 , l) in general. ( softmax(U ), t = 0 H1 (t) = H2 (t − 1), 0 < t ≤ T, Compatibility transform step can be viewed as another convolution layer where the spatial receptive field of the filter is 1 × 1, and the number of input and output channels are both L. Learning the weights of this filter is equivalent to learning the label compatibility function µ. Transferring error differentials from the output of this step to the input can be done since this step is a usual convolution operation. H2 (t) = fθ (U, H1 (t), I), 0 ≤ t ≤ T, ( 0, 0≤t<T Y (t) = H2 (t), t = T. (3) (4) (5) We name this RNN structure CRF-RNN. Parameters of the CRF-RNN are the same as the mean-field parameters described in Section 4 and denoted by θ here. Since the calculation of error differentials w.r.t. these parameters in a single iteration was described in Section 4, they can be learnt in the RNN setting using the standard back-propagation through time algorithm [48, 40]. It was shown in [29] that the mean-field iterative algorithm for dense CRF converges in less than 10 iterations. Furthermore, in practice, after about 5 iterations, increasing the number of iterations usually does not significantly improve results [29]. Therefore, 4.5. Adding Unary Potentials In this step, the output from the compatibility transform stage is subtracted element-wise from the unary inputs U . While no parameters are involved in this step, transferring error differentials can be done trivially by copying the differentials at the output of this step to both inputs with the appropriate sign. 5 segmentation of the image. We used the FCN-8s architecture of [37] as the first part of our network, which provides unary potentials to the CRF. This network is based on the VGG-16 network [53] but has been restructured to perform pixel-wise prediction instead of image classification. In the forward pass through the network, once the computation enters the CRF-RNN after passing through the CNN stage, it takes T iterations for the data to leave the loop created by the RNN. Neither the CNN that provides unary values nor the layers after the CRF-RNN (i.e., the loss layers) need to perform any computations during this time since the refinement happens only inside the RNN’s loop. Once the output Y leaves the loop, next stages of the deep network after the CRF-RNN can continue the forward pass. In our setup, a softmax loss layer directly follows the CRF-RNN and terminates the network. During the backward pass, once the error differentials reach the CRF-RNN’s output Y , they similarly spend T iterations within the loop before reaching the RNN input U in order to propagate to the CNN which provides the unary input. In each iteration inside the loop, error differentials are computed inside each component of the mean-field iteration as described in Section 4. We note that unnecessarily increasing the number of mean-field iterations T could potentially result in the vanishing and exploding gradient problems in the CRF-RNN. We, however, did not experience this problem during our experiments. I Meanfield Iteration U H2 = fθ (U, H1 , I) H1 H2 G2 Y G1 Softmax Normalization Figure 2. The CRF-RNN Network. We formulate the iterative mean-field algorithm as a Recurrent Neural Network (RNN). Gating functions G1 and G2 are fixed as described in the text. FCN CRF-RNN 6. Implementation Details In the present section we describe the implementation details of the proposed network, as well as its training process. The high-level architecture of our system, which was implemented using the popular Caffe [27] deep learning library, is shown in Fig. 3. The full source code and the trained models of our approach will be made publicly available 1 . We initialized the first part of the network using the publicly available weights of the FCN-8s network [37]. The compatibility transform parameters of the CRF-RNN were initialized using the Potts model, and kernel width and weight parameters were obtained from a cross-validation process. We found that such initialization results in faster convergence of training. During the training phase, parameters of the whole network were optimized end-to-end using the back-propagation algorithm. In particular we used full image training described in [37], with learning rate fixed at 10−13 and momentum set to 0.99. These extreme values of the parameters were used since we employed only one image per batch to avoid reaching memory limits of the GPU. In all our experiments, during training, we set the number of mean-field iterations T in the CRF-RNN to 5 to avoid Figure 3. The End-to-end Trainable Network. Schematic visualization of our full network which consists of a CNN and the CNN-CRF network. Best viewed in colour. it does not suffer from the vanishing and exploding gradient problem inherent to deep RNNs [7, 43]. This allows us to use a plain RNN architecture instead of more sophisticated architectures such as LSTMs in our network. 5.2. Completing the Picture Our approach comprises a fully convolutional network stage, which predicts pixel-level labels without considering structure, followed by a CRF-RNN stage, which performs CRF-based probabilistic graphical modelling for 1 structured prediction. The complete system, therefore, unifies strengths of both CNNs and CRFs and is trainable end-to-end using the back-propagation algorithm [34] and the Stochastic Gradient Descent (SGD) procedure. During training, a whole image (or many of them) can be used as the mini-batch and the error at each pixel output of the network can be computed using an appropriate loss function such as the softmax loss with respect to the ground truth 1 https://github.com/torrvision/crfasrnn/. 6 vanishing/exploding gradient problems and to reduce the training time. During the test time, iteration count was increased to 10. The effect of this parameter value on the accuracy is discussed in section 7.1. Loss function During the training of the models that achieved the best results reported in this paper, we used the standard softmax loss function, that is, the log-likelihood error function described in [30]. The standard metric used in the Pascal VOC challenge is the average intersection over union (IU), which we also use here to report the results. In our experiments we found that high values of IU on the validation set were associated to low values of the averaged softmax loss, to a large extent. We also tried the robust loglikelihood in [30] as a loss function for CRF-RNN training. However, this did not result in increased accuracy nor faster convergence. Normalization techniques As described in Section 4, we use the exponential function followed by pixel-wise normalization across channels in several stages of the CRFRNN. Since this operation has a tendency to result in small gradients with respect to the input when the input value is large, we conducted several experiments where we replaced this by a rectifier linear unit (ReLU) operation followed by a normalization across the channels. Our hypothesis was that this approach may approximate the original operation adequately while speeding up the training due to improved gradients. Furthermore, ReLU would induce sparsity on the probability of labels assigned to pixels, implicitly pruning low likelihood configurations, which could have a positive effect. However, this approach did not lead to better results, obtaining 1% IU lower than the original setting performance. Input Image FCN-8s B-ground Aero plane Bicycle Car Motorbike Cat Person DeepLab Chair Bird Cow Potted-Plant Sheep CRF-RNN Boat Dining-Table Sofa Ground Truth Bus Bottle Dog Horse Train TV/Monitor Figure 4. Qualitative results on the validation set of Pascal VOC 2012. FCN [37] is a CNN-based model that does not employ CRF. Deeplab [10] is a two-stage approach, where the CNN is trained first, and then CRF is applied on top of the CNN output. Our approach is an end-to-end trained system that integrates both CNN and CRF-RNN in one deep network. Best viewed in colour. 7. Experiments amounts to a total of 11,685 images. After removing the overlapping images between VOC 2012 validation data and this training dataset, we were left with 346 images from the original VOC 2012 validation set to validate our models on. We call this set the reduced validation set in the sequel. Annotations of the VOC 2012 test set, which consists of 1456 images, are not publicly available and hence the final results on the test set were obtained by submitting the results to the Pascal VOC challenge evaluation server [18]. Regardless of the smaller number of images, we found that the relative improvements of the accuracy on our validation set were in good agreement with the test set. We present experimental results with the proposed CRFRNN framework. We use these datasets: the Pascal VOC 2012 dataset, and the Pascal Context dataset. We use the Pascal VOC 2012 dataset as it has become the golden standard to comprehensively evaluate any new semantic segmentation approach in comparison to existing methods. We also use the Pascal Context dataset to assess how well our approach performs on a dataset with different characteristics. Pascal VOC Datasets As a first step we directly compared the potential advantage of learning the model end-to-end with respect to alternative learning strategies. These are plain FCN-8s without applying CRF, and with CRF as a postprocessing method disconnected from the training of FCN, which is comparable to the approach described in [10] and [41]. The results are reported in Table 1 and show a clear advantage of the end-to-end strategy over the offline application of CRF as a In order to evaluate our approach with existing methods under the same circumstances, we conducted two main experiments with the Pascal VOC 2012 dataset, followed by a qualitative experiment. In the first experiment, following [37, 38, 41], we used a training set consisted of VOC 2012 training data (1464 images), and training and validation data of [23], which 7 Method Plain FCN-8s FCN-8s and CRF disconnected End-to-end training of CRF-RNN post-processing method. This can be attributed to the fact that during the SGD training of the CRF-RNN, the CNN component and the CRF component learn how to co-operate with each other to produce the optimum output of the whole network. We then proceeded to compare our approach with all state-of-the-art methods that used training data from the standard VOC 2012 training and validation sets, and from the dataset published with [22]. The results are shown in Table 2, above the bar, and we can see that our approach outperforms all competitors. In the second experiment, in addition to the above training set, we used data from the Microsoft COCO dataset [36] as was done in [41] and [12]. We selected images from MS COCO 2014 training set where the ground truth segmentation has at least 200 pixels marked with classes labels present in the VOC 2012 dataset. With this selection, we ended up using 66,099 images from the COCO dataset and therefore a total of 66,099 + 11,685 = 77,784 training images were used in the second experiment. The same reduced validation set was used in this second experiment as well. In this case, we first fine-tuned the plain FCN-32s network (without the CRF-RNN part) on COCO data, then we built an FCN-8s network with the learnt weights and finally train the CRF-RNN network end-to-end using VOC 2012 training data only. Since the MS COCO ground truth segmentation data contains somewhat coarse segmentation masks where objects are not delineated properly, we found that fine-tuning our model with COCO did not yield significant improvements. This can be understood because the primary advantage of our model comes from delineating the objects and improving fine segmentation boundaries. The VOC 2012 training dataset therefore helps our model learn this task effectively. The results of this experiment are shown in Table 2, below the bar, and we see that our approach sets a new state-of-the-art on the VOC 2012 dataset. Note that in both setups, our approach outperforms competing methods due to the end-to-end training of the CNN and CRF in the unified CRF-RNN framework. We also evaluated our models on the VOC 2010, and VOC 2011 test set (see Table 2). In all cases our method achieves the stateof-the-art performance. In order to have a qualitative evidence about how CRFRNN learns, we visualize the compatibility function learned after the training stage of the CRF-RNN as a matrix representation in Fig. 5. Element (i, j) of this matrix corresponds to µ(i, j) defined earlier: a high value at (i, j) implies high penalty for assigning label i to a pixel when a similar pixel (spatially or appearance wise) is assigned label j. For example we can appreciate that the learned compatibility matrix assigns a low penalty to pairs of labels that tend to appear together, such as [Motorbike, Person], and [Dining table, Chair]. Without COCO 61.3 With COCO 68.3 63.7 69.5 69.6 72.9 Table 1. Mean IU accuracy of our approach, CRF-RNN, compared with similar methods, evaluated on the reduced VOC 2012 validation set. Method BerkeleyRC [3] O2PCPMC [8] Divmbest [44] NUS-UDS [16] SDS [23] MSRACFM [13] FCN-8s [37] Hypercolumn [24] Zoomout [38] Context-DeepCNN-CRF [35] DeepLabMSc [10] Our method w/o COCO BoxSup [12] DeepLab [10, 41] Our method with COCO VOC 2010 test n/a 49.6 n/a n/a n/a VOC 2011 test 39.1 48.8 n/a n/a n/a VOC 2012 test n/a 47.8 48.1 50.0 51.6 n/a n/a 61.8 n/a n/a 64.4 62.7 n/a 64.1 62.2 62.6 64.4 n/a n/a 70.7 n/a n/a 71.6 73.6 72.4 72.0 n/a n/a 71.0 n/a n/a 72.7 75.7 75.0 74.7 Table 2. Mean IU accuracy of our approach, CRF-RNN, compared to the other approaches on the Pascal VOC 2010-2012 test datasets. Methods from the first group do not use MS COCO data for training. The methods from the second group use both COCO and VOC datasets for training. Pascal Context Dataset We conducted an experiment on the Pascal Context dataset [39], which differs from the previous one in the larger number of classes considered, 59. We used the provided partitions of training and validation sets, and the obtained results are reported in Table 3. FCNCRF8s [37] RNN Mean IU 18.1 34.4 37.78 39.28 Table 3. Mean IU accuracy of our approach, CRF-RNN, evaluated on the Pascal Context validation set. Method 8 O2 P [8] CFM [13] B-Ground Aeroplane Bicycle Bird Boat Bottle Bus Car Cat Chair Cow Dining-Table Dog Horse Motorbike Person PottlePlant Sheep Sofa Train TV/Monitor to-end with 5 such iterations yielded a final mean IU score of only 70.9, supporting the hypothesis that the recurrent structure of our approach is important for its success. 0 0.0 -0.1 -0.2 8. Conclusion -0.3 -0.4 We presented CRF-RNN, an interpretation of dense CRFs as Recurrent Neural Networks. Our formulation fully integrates CRF-based probabilistic graphical modelling with emerging deep learning techniques. In particular, the proposed CRF-RNN can be plugged in as a part of a traditional deep neural network: It is capable of passing on error differentials from its outputs to inputs during back-propagation based training of the deep network while learning CRF parameters. We demonstrate the use of this approach by utilizing it for the semantic segmentation task: we form an end-to-end trainable deep network by combining a fully convolutional neural network with the CRF-RNN. Our system achieves a new state-of-the-art on the popular Pascal VOC segmentation benchmark. This improvement can be attributed to the uniting of the strengths of CNNs and CRFs in a single deep network. In the future, we plan to investigate the advantages/disadvantages of restricting the capabilities of the RNN part of our network to mean-field inference of dense CRF. A sensible baseline to the work presented here would be to use more standard RNNs (e.g. LSTMs) that learn to iteratively improve the input unary potentials to make them closer to the ground-truth. -0.5 -0.5 -0.6 -0.7 -0.8 -0.9 B-Ground Aeroplane Bicycle Bird Boat Bottle Bus Car Cat Chair Cow Dining-Table Dog Horse Motorbike Person PottlePlant Sheep Sofa Train TV/Monitor -1.0 Figure 5. Visualization of the learnt label compatibility matrix. In the standard Potts model, diagonal entries are equal to −1, while off-diagonal entries are zero. These values have changed after the end-to-end training of our network. Best viewed in colour. 7.1. Effect of Design Choices We performed a number of additional experiments on the Pascal VOC 2012 validation set described above to study the effect of some design choices we made. We first studied the performance gains attained by our modifications to the CRF over the CRF approach proposed by [29]. We found that using different filter weights for different classes improved the performance by 1.8 percentage points, and that introducing the asymmetric compatibility transform further boosted the performance by 0.9 percentage points. Regarding the RNN parameter iteration count T , incrementing it to T = 10 during the test time, from T = 5 during the train time, produced an accuracy improvement of 0.2 percentage points. Setting T = 10 also during training reduced the accuracy by 0.7 percentage points. We believe that this might be due to a vanishing gradient effect caused by using too many iterations. In practice that leads to the first part of the network (the one producing unary potentials) receiving a very weak error gradient signal during training, thus hampering its learning capacity. End-to-end training after the initialization of CRF parameters improved performance by 3.4 percentage points. We also conducted an experiment where we froze the FCN8s part and fine-tuned only the RNN part (i.e., CRF parameters). It improved the performance over initialization by only 1 percentage point. We therefore conclude that end-toend training significantly contributed to boost the accuracy of the system. Treating each iteration of mean-field inference as an independent step with its own parameters, and training end- Acknowledgement This work was supported by grants EP/M013774/1 and ERC 321162-HELIOS. We thank the Caffe team, Baidu IDL, and the Oxford ARC team for their support. We gratefully acknowledge GPU donations from NVIDIA. References [1] A. Adams, J. Baek, and M. A. Davis. Fast highdimensional filtering using the permutohedral lattice. Computer Graphics Forum, 29(2):753–762, 2010. [2] P. Arbeláez, B. Hariharan, C. Gu, S. Gupta, L. Bourdev, and J. Malik. Semantic segmentation using regions and parts. In IEEE CVPR, 2012. [3] P. Arbeláez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. IEEE TPAMI, 33(5):898–916, 2011. [4] A. Barbu. Training an active random field for realtime image denoising. IEEE TIP, 18(11):2451–2462, 2009. [5] S. Bell, P. Upchurch, N. Snavely, and K. Bala. Material recognition in the wild with the materials in context database. In IEEE CVPR, 2015. 9 [6] Y. Bengio, Y. LeCun, and D. Henderson. Globally trained handwritten word recognizer using spatial representation, convolutional neural networks, and hidden markov models. In NIPS, pages 937–937, 1994. [7] Y. Bengio, P. Simard, and P. Frasconi. Learning longterm dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 1994. [8] J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu. Free-form region description with second-order pooling. IEEE TPAMI, 2014. [9] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In BMVC, 2014. [10] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015. [11] L.-C. Chen, A. G. Schwing, A. L. Yuille, and R. Urtasun. Learning deep structured models. In ICLRW, 2015. [12] J. Dai, K. He, and J. Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In arXiv:1503.01640, 2015. [13] J. Dai, K. He, and J. Sun. Convolutional feature masking for joint object and stuff segmentation. In IEEE CVPR, 2015. [14] T.-M.-T. Do and T. Artieres. Neural conditional random fields. In NIPS, 2010. [15] J. Domke. Learning graphical model parameters with approximate marginal inference. IEEE TPAMI, 35(10):2454–2467, 2013. [16] J. Dong, Q. Chen, S. Yan, and A. Yuille. Towards unified object detection and semantic segmentation. In ECCV, 2014. [17] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS, 2014. [18] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111(1):98–136, 2015. [19] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. IEEE TPAMI, 2013. [20] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE CVPR, 2014. [21] R. Girshick, F. Iandola, T. Darrell, and J. Malik. Deformable part models are convolutional neural networks. In CVPR, 2015. [22] B. Hariharan, P. Arbelaez, L. D. Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In IEEE ICCV, 2011. [23] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV, 2014. [24] B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and finegrained localization. In IEEE CVPR, 2015. [25] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman. Deep structured output learning for unconstrained text recognition. In ICLR, 2015. [26] V. Jain, J. F. Murray, F. Roth, S. C. Turaga, V. P. Zhigulin, K. L. Briggman, M. Helmstaedter, W. Denk, and H. S. Seung. Supervised learning of image restoration with convolutional networks. In IEEE ICCV, 2007. [27] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM Multimedia, pages 675–678, 2014. [28] M. Kiefel and P. V. Gehler. Human pose estmation with fields of parts. In ECCV, 2014. [29] P. Krähenbühl and V. Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NIPS, 2011. [30] P. Krähenbühl and V. Koltun. Parameter learning and convergent inference for dense random fields. In ICML, 2013. [31] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. [32] L. Ladicky, C. Russell, P. Kohli, and P. H. Torr. Associative hierarchical crfs for object class image segmentation. In IEEE ICCV, 2009. [33] J. D. Lafferty, A. McCallum, and F. C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, 2001. [34] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [35] G. Lin, C. Shen, I. Reid, and A. van dan Hengel. Efficient piecewise training of deep structured models for semantic segmentation. In arXiv:1504.01013, 2015. [36] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollar. Microsoft coco: Common objects in context. In arXiv:1405.0312, 2014. 10 [37] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In IEEE CVPR, 2015. [51] J. Shotton, M. Johnson, and R. Cipolla. Semantic texton forests for image categorization and segmentation. In IEEE CVPR, 2008. [38] M. Mostajabi, P. Yadollahpour, and G. Shakhnarovich. Feedforward semantic segmentation with zoom-out features. In IEEE CVPR, 2015. [52] J. Shotton, J. Winn, C. Rother, and A. Criminisi. Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV, 81(1):2–23, 2009. [39] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE CVPR, 2014. [53] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In arXiv:1409.1556, 2014. [54] V. Stoyanov, A. Ropson, and J. Eisner. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In AISTATS, 2011. [55] S. C. Tatikonda and M. I. Jordan. Loopy belief propagation and gibbs measures. In Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, 2002. [40] M. C. Mozer. Backpropagation. In Y. Chauvin and D. E. Rumelhart, editors, Backpropagation, chapter A Focused Backpropagation Algorithm for Temporal Pattern Recognition, pages 137–169. L. Erlbaum Associates Inc., 1995. [41] G. Papandreou, L.-C. Chen, K. Murphy, and A. L. Yuille. Weakly- and semi-supervised learning of a dcnn for semantic image segmentation. In arXiv:1502.02734, 2015. [56] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In IEEE CVPR, 1998. [42] S. Paris and F. Durand. A fast approximation of the bilateral filter using a signal processing approach. IJCV, 81(1):24–52, 2013. [57] J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In NIPS, 2014. [43] R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. On the difficulty of training recurrent neural networks. In ICML, 2013. [58] Z. Tu. Auto-context and its application to high-level vision tasks. In IEEE CVPR, 2008. [59] Z. Tu, X. Chen, A. L. Yuille, and S.-C. Zhu. Image parsing: Unifying segmentation, detection, and recognition. IJCV, 63(2):113–140, 2005. [44] G. S. Payman Yadollahpour, Dhruv Batra. Discriminative re-ranking of diverse segmentations. In IEEE CVPR, 2013. [45] J. Peng, L. Bo, and J. Xu. Conditional neural fields. In NIPS, 2009. [60] K. Yao, B. Peng, G. Zweig, D. Yu, X. Li, and F. Gao. Recurrent conditional random field for language understanding. In ICASSP, 2014. [46] P. H. O. Pinheiro and R. Collobert. Recurrent convolutional neural networks for scene labeling. In ICML, 2014. [61] Y. Zhang and T. Chen. Efficient inference for fullyconnected crfs with stationarity. In CVPR, 2012. [47] S. Ross, D. Munoz, M. Hebert, and J. A. Bagnell. Learning message-passing inference machines for structured prediction. In IEEE CVPR, 2011. [48] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Parallel distributed processing: explorations in the microstructure of cognition. In J. A. Anderson and E. Rosenfeld, editors, Parallel distributed processing: explorations in the microstructure of cognition, chapter Learning Internal Representations by Error Propagation, pages 318–362. MIT Press, 1986. [49] A. G. Schwing and R. Urtasun. Fully connected deep structured networks. In arXiv:1503.02351, 2015. [50] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-time human pose recognition in parts from single depth images. In IEEE CVPR, 2011. 11 Methods trained with COCO Our method DeepLab[10, 41] BoxSup[12] Methods trained w/o COCO Our method trained w/o COCO DeepLab-MSc-CRF-LargeFOV[10] Context Deep CNN CRF[35] Zoomout[38] Hypercolumn[24] FCN-8s[37] MSRA CFM[13] SDS[23] NUS UDS [16] TTIC-divmbest-rerank[44] BONN O2PCPMC FGT SEGM [8] Methods trained with COCO Our method DeepLab[10, 41] BoxSup[12] Methods trained w/o COCO Our method trained w/o COCO DeepLab-MSc-CRF-LargeFOV [10] Context Deep CNN CRF[35] TTI zoomout 16[38] Hypercolumn[24] FCN-8s[37] MSRA CFM[13] SDS[23] NUS UDS[16] TTIC-divmbest-rerank[44] BONN O2PCPMC FGT SEGM[8] Mean IU 74.7 72.7 71.0 90.4 89.1 86.4 55.3 38.3 35.5 88.7 88.1 79.7 68.4 63.3 65.2 69.8 69.7 65.2 88.3 87.1 84.3 82.4 83.1 78.5 85.1 85.0 83.7 32.6 29.3 30.5 72.0 71.6 70.7 64.4 62.6 62.2 61.8 51.6 50.0 48.1 47.8 87.5 84.4 87.5 81.9 68.7 76.8 75.7 63.3 67.0 62.7 64.0 39.0 54.5 37.7 35.1 33.5 34.2 26.7 25.7 24.5 25.6 27.3 79.7 81.5 75.8 78.2 69.8 68.9 69.5 63.0 47.2 46.9 54.1 64.2 63.6 57.4 57.4 51.3 49.4 48.8 39.8 45.0 43.0 39.2 68.3 65.9 72.3 56.5 70.2 60.3 65.6 59.2 47.9 54.8 48.7 87.6 85.1 88.4 80.5 81.1 75.3 81.0 70.9 65.3 58.4 56.6 80.8 79.1 82.6 74.0 71.9 74.7 69.2 61.4 60.6 58.6 57.7 84.4 83.4 80.0 79.8 74.9 77.6 73.3 54.9 58.5 55.6 52.5 30.4 30.7 33.4 22.4 23.9 21.4 30.0 16.8 15.5 14.6 14.2 78.5 76.5 76.2 64.4 56.5 62.6 79.6 79.8 79.3 81.9 77.9 76.1 86.4 85.8 82.1 81.8 82.4 81.3 58.6 57.4 57.0 82.4 84.3 78.2 53.5 54.9 55.0 77.4 80.5 72.5 70.1 64.1 68.1 78.2 74.1 71.5 69.6 60.6 62.5 68.7 45.0 50.8 47.5 54.8 60.4 59.8 55.0 53.7 46.9 46.8 51.5 48.2 37.4 31.2 29.6 80.5 79.0 79.3 74.0 72.1 71.8 69.1 50.5 45.8 44.7 42.2 77.8 76.1 78.4 76.0 68.3 63.9 68.1 51.0 59.9 51.0 58.0 83.1 83.2 81.3 76.6 74.5 76.5 71.7 57.7 62.0 60.9 54.8 80.6 80.8 82.7 68.8 72.9 73.9 67.5 63.3 52.7 53.5 50.2 59.5 59.7 56.1 44.3 52.6 45.2 50.4 31.8 40.8 36.6 36.6 82.8 82.2 79.8 70.2 64.4 72.4 66.5 58.7 48.2 50.9 58.6 47.8 50.4 48.6 40.2 45.4 37.4 44.4 31.2 36.8 30.1 31.6 78.3 73.1 77.1 68.9 64.9 70.9 58.9 55.7 53.1 50.2 48.4 67.1 63.7 66.3 55.3 57.4 55.1 53.5 48.5 45.6 46.8 38.6 Table 4. Intersection over Union (IU) accuracy of our approach, CRF-RNN, compared to the other state-of-the-art approaches on the Pascal VOC 2012 test set. Scores for other methods were taken the results published by the original authors. The symbols are from Chatfield et al. [9]. 12 TV/Monitor Sofa Sheep Potted-Plant Motorbike Car Cat Person Chair Dining-Table Bottle Dog B-ground Aero plane Bicycle Boat Bird Cow Train Ground Truth Horse CRF-RNN Bus Input Image Figure 6. Typical good quality segmentation results I. Illustration of sample results on the validation set of the Pascal VOC 2012 dataset. Note that in some cases our method is able to pick correct segmentations that are not marked correctly in the ground truth. Best viewed in colour. 13 TV/Monitor Sofa Sheep Potted-Plant Motorbike Car Cat Person Chair Dining-Table Bottle Dog B-ground Aero plane Bicycle Boat Bird Cow Train Ground Truth Horse CRF-RNN Bus Input Image Figure 7. Typical good quality segmentation results II. Illustration of sample results on the validation set of the Pascal VOC 2012 dataset. Note that in some cases our method is able to pick correct segmentations that are not marked correctly in the ground truth. Best viewed in colour. 14 TV/Monitor Sofa Sheep Potted-Plant Motorbike Car Cat Person Chair Dining-Table Bottle Dog B-ground Aero plane Bicycle Boat Bird Cow Train Ground Truth Horse CRF-RNN Bus Input Image Figure 8. Failure cases I. Illustration of sample failure cases on the validation set of the Pascal VOC 2012 dataset. Best viewed in colour. 15 TV/Monitor Sofa Sheep Potted-Plant Motorbike Car Cat Person Chair Dining-Table Bottle Dog B-ground Aero plane Bicycle Boat Bird Cow Train Ground Truth Horse CRF-RNN Bus Input Image Figure 9. Failure cases II. Illustration of sample failure cases on the validation set of the Pascal VOC 2012 dataset. Best viewed in colour. 16 Input Image FCN-8s Bicycle Aero plane B-ground B-ground Bicycle Aero plane Car Chair Bird Bird Cow Cow Cat Chair Cat Potted-Plant Person Motorbike Sheep Potted-Plant Person Motorbike Sheep Car CRF-RNN DeepLab Boat Boat Dinging-table Bus BottleBottle Horse Dog Dining-Table Sofa Sofa Ground Truth Train Bus Horse Dog TV/Monitor Train TV/Monitor Figure 10. Qualitative comparison with the other approaches. Sample results with our method on the validation set of the Pascal VOC 2012 dataset, compared with previous state-of-the-art methods. Segmentation results with DeepLap approach were reproduced from the original publication. Best viewed in colour. 17 Local Convolutional Features with Unsupervised Training for Image Retrieval Mattis Paulin, Matthijs Douze, Zaid Harchaoui, Julien Mairal, Florent Perronnin, Cordelia Schmid To cite this version: Mattis Paulin, Matthijs Douze, Zaid Harchaoui, Julien Mairal, Florent Perronnin, et al.. Local Convolutional Features with Unsupervised Training for Image Retrieval. ICCV - IEEE International Conference on Computer Vision, Dec 2015, Santiago, Chile. pp.91-99, �10.1109/ICCV.2015.19�. �hal01207966� HAL Id: hal-01207966 https://hal.inria.fr/hal-01207966 Submitted on 1 Oct 2015 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Local Convolutional Features with Unsupervised Training for Image Retrieval Mattis Paulin 1 Julien Mairal1 1 Inria ∗ Matthijs Douze1 Florent Perronnin3 2 Zaid Harchaoui1,2 Cordelia Schmid1 3 NYU Facebook AI Research† Abstract Patch-level descriptors underlie several important computer vision tasks, such as stereo-matching or contentbased image retrieval. We introduce a deep convolutional architecture that yields patch-level descriptors, as an alternative to the popular SIFT descriptor for image retrieval. The proposed family of descriptors, called PatchCKN, adapt the recently introduced Convolutional Kernel Network (CKN), an unsupervised framework to learn convolutional architectures. We present a comparison framework to benchmark current deep convolutional approaches along with Patch-CKN for both patch and image retrieval, including our novel “RomePatches” dataset. Patch-CKN descriptors yield competitive results compared to supervised CNN alternatives on patch and image retrieval. Figure 1. Proposed approach: interest regions are first extracted from images; then the neighborhood of each point is affine- and orientation-normalized to sample a patch; finally, deep convolutional nets are used to get a descriptor for each patch. 1. Introduction or to small rotations.1 As for step 3), the goal is to define a suitable metric between two patch sets. To avoid the cost of matching individual patches, scalable alternatives have been proposed that encode and aggregate local patch statistics, such as bag-of-words [41] and VLAD [19]. In this work, we focus on step 2), i.e. the description step, while we rely on state-of-the-art components for the detection and matching steps. Our inspiration comes from the expressive feature representations output by deep convolutional neural nets (CNNs) [23] used in image classification [22]. Features output by a CNN’s intermediate layers can be used as imagelevel descriptors [10] which can be transferred to a variety of tasks – see e.g. [3, 13, 31]. More recently, the question of whether suitable patch-level descriptors could be derived from such architectures has been raised. [25, 12, 39] provide a preliminary positive answer to this question by comparing favorably with the SIFT descriptor. While these works exhibit significant differences, it is worth noting they This paper introduces a deep kernel-based convolutional approach for the description of image patches that does not require supervision. The kernel-based feature representation can be effectively approximated using a simple stochastic gradient optimization procedure, yielding a patch-level descriptor that can be used for image retrieval tasks. Image retrieval is a challenging problem as different images of the same object/scene may exhibit large variations in viewpoint, illumination, scaling, occlusion, etc – see Figure 1. State-of-the-art instance-level retrieval systems involve three steps: 1) interest point detection, 2) description and 3) matching. The goal of step 1) is to select key points that are reproducible under scale and viewpoint changes – see [30, 43] for a detailed comparison of detectors. The choice of a good local representation at step 2) is crucial to ensure robustness to viewing conditions. As an example, the popular SIFT descriptor [26] is robust to illumination variations ∗ LEAR team, Inria Grenoble Rhône-Alpes, Laboratoire Jean Kuntzmann, CNRS, Univ. Grenoble Alpes, France. † Research conducted while at the Xerox Research Centre Europe. 1 SIFT 1 refers only to the description part. all rely on supervised learning processes. While the penultimate layer outputs a suitable imagelevel descriptor [3, 13, 31], the output of previous layers, typically the 4th one, is actually preferable to obtain expressive patch-level descriptors, as noticed in [25]. As shown in [47], earlier layers tend to encode more task-independent information with respect to the later ones. In particular, the filters learned by the first layer tend to be similar regardless of the task, the objective function or the level of supervision. This motivates the following question: is supervised learning required to make good local convolutional features for patch matching and image retrieval? Our contribution is the family of patch descriptors Patch-CKN based on Convolutional Kernel Networks (CKNs) [27]. CKNs were initially introduced for image classification in [27]. We introduce a feature representation for patches that is based on the kernel feature map of a convolutional match kernel, and therefore does not depend on examples nor labels. A finite-dimensional explicit feature embedding can be computed to approximate this kernel feature map [34, 44, 27]. We present a fast and simple procedure to compute an explicit feature embedding, using random sub-sampling of the patches, suitable for largescale experiments. The experiments suggest that PatchCKN gives competitive patch-level descriptors compared to supervised CNNs. Several works have focused on learning patch representations [7, 46], yet few have analyzed the impact of improving patch retrieval on image retrieval performance. For this purpose, we introduce a new dataset, “RomePatches”, using images and the 3D reconstruction of [1]. The 16K Flickr images of “RomePatches” represent views of 66 different locations in Rome. The 3D reconstruction provides sparse patch matches, yielding the ground truth for our patch retrieval dataset. This allows relating performance improvements for both patch and image retrieval tasks. In a nutshell, our main contributions are three-fold: we describe our image retrieval pipeline, and devote Sec. 4 to convolutional descriptors. Our new dataset is introduced in Sec. 5, and experimental results are presented in Sec. 6. Our new dataset as well as the code to extract Patch-CKN are available online2 . 2. Related Work Our literature review focuses on the works which are closest to ours: shallow patch descriptors, deep learning for image retrieval and deep learning for patch description. Traditional patch descriptors. Among the variety of standard patch descriptors, SIFT [26] is the most widely used. Interpreted as a convolutional net, SIFT is a two-layer architecture, the first layer computing patch gradient orientations, average-pooled in the second one. SIFT has been successfully used for many tasks such as stereo matching [1], content-based retrieval [16], or classification [33]. Mikolajczyk et al. [29] provide a detailed survey of local descriptors and demonstrate the excellent performance of SIFT. Improved local descriptors include BRIEF [8] and LIOP [45]. All these descriptors are hand-crafted and have been optimized by grid-search on a relatively small amount of parameters. When the number of parameters to be set is large, such as approach is infeasible and the optimal parametrization needs to be learned from data. Most works on hand-crafted descriptor learning use supervision. Brown et al. [7, 46] designed a matching dataset from reprojected 3D models of landmarks, obtained by structure from motion, with a descriptor consisting of several existing parts, including but not limited to SIFT, GLOH [29] and Daisy [42]. We do not include their dataset in our experiments, because of significant differences in early stages of the pipeline as the Multi-view Stereo Correspondence Dataset contains few images of just three locations, with grey-scale patches extracted (while in this work we leverage the additional color information), that were extracted with a detector that is incompatible with ours (DoG instead of Hessian-affine). Philbin et al. [37] learn a Mahalanobis metric for SIFT descriptors to compensate the binarization error, with excellent results in instance-based retrieval. Simonyan et al. [40] propose the “Pooling Regions” descriptor and learn its parameters, as well as a linear projection using stochastic optimization. Their learning objective can be cast as a convex optimization problem, which is not the case for classical convolutional networks. An exception is [5] which presents a match-kernel interpretation of SIFT, and a family of kernel descriptors whose parameters are learned in an unsupervised fashion. The Patch-CKN we introduce generalizes kernel descriptors; the proposed procedure for computing an explicit feature embedding is faster and simpler. 1. We propose a patch descriptor based on a CKN architecture [27], using a fast and simple stochastic procedure to compute an explicit feature embedding. 2. We introduce and make available a dataset, named “Rome-Patches”, for the evaluation of patch and image retrieval, enabling to systematically study the correlation between patch matching and image retrieval performance. 3. We show that, for the purpose of patch and image retrieval, it is possible to learn competitive patch-level descriptors without supervision, and therefore at a fraction of the computational and annotation cost compared to previous supervised alternatives [25, 11]. 2 lear.inrialpes.fr/people/paulin/projects/ RomePatches Overview. We review related work in Sec. 2. In Sec. 3, 2 descriptor SIFT [26] Daisy [46] AlexNet [22] Neural codes [3] PhilippNet [12] Fracking [39] CKN [27] application sparse features patch matching image classification same-image recognition patch matching patch matching image classification supervision N/A class = 3D location object classes landmark images artificial classes match/non-match no supervision #parameters 2 0 0 10 50 70M 50 70M 10 10k 10 46k 10 256k optimization method N/A Powel’s conjugate direction method backpropagation, SGD fine-tuning on top of AlexNet backpropagation, SGD backpropagation, SGD layer-wise SGD Table 1. Levels of supervision and optimization methods used by the approaches related to this work. There are two columns for parameters: hyper-parameters (tuned by hand) and parameters determined by the optimization method. 3. Image Retrieval Pipeline Deep learning for image retrieval. With a CNN learned on a sufficiently large labeled set such as ImageNet [9], the output of its intermediate layers can be used as image descriptors for a wide variety of tasks including image retrieval [3, 38] – the focus of this work. The output of one of the fully-connected layers is often chosen because it is compact, usually 4,096 D. However, global CNN descriptors lack geometric invariance [14], so they produce results below the state-of-the-art in instance-level image retrieval. Hence, improvements have been proposed. We briefly present the three-step pipeline: interest point detection, patch description, and patch matching. Interest point detection. Interest point detectors provide locations invariant to certain image transformations. This ensures that two views of the same scene even with changes in viewpoint or illumination share similar “interest points”, see [30] for a review of detectors. We use the popular Hessian-Affine detector [28]. The idea is to extract points at their characteristic scale and estimate for each point an affine-invariant local region, see Fig. 1. Rotation invariance is obtained by rotating patches to align the dominant gradient orientation. This results in a set of interest points associated with locally affine-invariant regions. Interest point description. Given a normalized patch M obtained by mapping the affine region to a fixed-size square, we compute its feature representation φ(M ) in a Euclidean space. The representation is expected to be robust to the perturbations that are not covered by the detector (lighting changes, small rotations, blur,...). Patch matching. Because matching all possible pairs of patches is too expensive, we follow the standard practice of encoding the patch descriptors and aggregating them into a fixed-length image descriptor, using the VLAD representation [18]. Given a clustering of the feature space consisting of k centroids {c1 , . . . , ck }, VLAD encodes a set of descriptors as the total shift with respect to their assigned centroid. A power normalization with exponent 0.5 is then applied to the VLAD descriptor, as well as an L2 normalization. In [38, 14], CNN responses at different scales and positions are extracted. We proceed similarly, yet we replace the (coarse) dense grid with a patch detector. There are important differences between [38, 14] and our work. While [14, 38] use the output of the penultimate layer as patch descriptor, we show in our experiments that we can get improved results with the output of preceding layers, that are cheaper to compute. In [3], the authors use a single global CNN descriptor for instance-based image retrieval and finetune the descriptor on a surrogate landmark dataset. While fine-tuning improves results, it would be difficult to replicate this success beyond landmarks. Finally, [21] proposes a Siamese architecture to train image retrieval descriptors but does not report results on standard retrieval benchmarks. Deep patch descriptors. Recently [25, 12, 39] reported superior results to SIFT for tasks such as patch matching or patch classification. The three works use different levels of supervision to train a CNN: category labels in [25], surrogate patch labels in [12] (each class is a given patch under different transformations) and matching/non-matching pairs in [39]. There are two key differences between those works and ours. First, they focus on patch-level metrics, instead of actual image retrieval. Second, and more importantly, while all these approaches require some kind of supervision, we show that our Patch-CKN yields competitive performance in both patch matching and image retrieval without requiring supervision. Especially, with respect to [25, 39] we do not need costly labels. And compared to [12] we do not need to make arbitrary choices in the definition of classes (i.e. the set of transformations). Table 1 summarizes the competing approaches. 4. Convolutional Descriptors We use convolutional features to encode fixed-size image patches (size 51×51 pixels). CNNs are normally trained with class supervision for a classification task. This can be extended to image retrieval by either: (i) encoding local descriptors with a model that has been trained for an unrelated image classification task, see Section 4.1; (ii) devising a surrogate classification problem that is as related as possible to image retrieval; (iii) using unsupervised learning, such as a convolutional kernel network, see Sec. 4.2. 3 Transformed versions of the same patch share the same label, thus defining surrogate classes. In this paper, we evaluate this strategy by using the same architecture and filter values, called PhilippNet, as in [12]. The network has three convolutional and one fully connected layers, takes as input 64x64 patches, and produces a 512-dimensional output. Mk+1 (z1 ) feature pooling γk Mk0 (z00 ) convolution Wk + non-linearity σk 4.2. Convolutional Kernels Networks CKNs have the same architecture as classical CNNs presented in Eq. (1) and in Fig. 2. The feature representation of CNNs relies on filters that are learned and hence defined in a data-dependent manner. We define here a feature representation that is based on a kernel (feature) map. The exact version of this feature representation is therefore data-independent. An explicit kernel (feature) map can be computed [34, 44, 27] to approximate it for computational efficiency. We present here a fast and simple procedure for this purpose, using sub-sampling of patches and stochastic gradient optimization, yielding a CKN that outputs patch descriptors. Let M and M 0 be two patches of size m × m (m = 51 in this paper), and Ω = {1, . . . , m}2 be the set of pixel locations. Let us also consider a fixed sub-patch size and denote by pz the sub-patch from M (resp. p0z the sub-patch from M 0 ) centered at location z ∈ Ω.3 Single-layer kernel definition. We consider the following kernel [27]: X 0 2 2 K1 (M, M 0 ) = e−kz−z k /2β1 k1 (pz , p0z0 ), (2) patch pz Mk (z0 ) Figure 2. A typical organization for two successive layers of a CNN. The spatial map Mk0 is obtained from Mk by convolution and pointwise non-linearity, and the top layer Mk+1 is obtained from Mk0 by a downsampling operation called feature pooling. By convention the map M0 corresponds to the input image x. 4.1. Convolutional Neural Networks Convolutional neural nets transform an input image by a sequence of simple operations or layers. Each layer performs a linear operation followed by a pointwise nonlinearity. Formally, the output f (x) of a CNN for some image x represented as a vector is f (x) = γK (σK (WK . . . γ2 (σ2 (W2 γ1 (σ1 (W1 x)) . . .)), (1) where the terms Wk are matrices corresponding to linear operations, the functions σk are pointwise non-linear functions, e.g., sigmoids or rectified linear units, and the functions γk perform a downsampling operation (feature pooling). For a convolutional layer, the matrices Wk have a particular structure and correspond to convolutions of spatial maps, as illustrated in Fig. 2. When they are dense and unstructured, the layer is called “fully connected”. z,z 0 ∈Ω where 0 k1 (pz , p0z0 ) = kpz kkp0z0 ke−kp̃z −p̃z0 k 2 /2α21 , (3) α1 and β1 are two kernel hyperparameters, k · k denotes the usual L2 norm, and p̃z and p̃0z0 are L2 -normalized versions of the sub-patches pz and p0z0 . The corresponding kernel (feature) map defines a feature representation for patches and images. Furthermore, the kernel is a match kernel. Therefore, the kernel offers a tunable level of invariance through the choice of hyperparameters, and produces hierarchical convolutional representations that are well-suited for natural images. Kernel embedding approximation. Since the exact computation of (2-3) is overwhelming, Mairal et al. propose an explicit finite-dimensional embedding [34, 44] to approximate it. The embedding of [27] keeps the 2-D spatial structure, similar to CNN feature maps. For the one-layer CKN, the approximation of [27] is: X K1 (M, M 0 ) ≈ g1 (u; M )T g1 (u; M 0 ) Learning from category labels. The most popular off-theshelf CNN is AlexNet [22], which won the ImageNet 2012 challenge. AlexNet has 7 layers: the first five are convolutional and the last ones are fully connected. The network is designed to process images of size 224 × 224, but convolutional layers may be fed with smaller inputs to produce 1x1 maps that we can use as low-dimensional patch descriptors – see the “coverage” column in Table 3. To ensure a fair comparison between all approaches, we rescale patches to always produce a 1 × 1 map. Learning from surrogate labels. Most CNNs such as AlexNet augment the dataset with perturbed versions of training patches to learn the filters Wk in (1). The authors of [11, 12] use “virtual patches”, obtained as transformations of randomly extracted ones to fall back to a classification problem. For a set of patches P, and a set a transformations T , the dataset consists of all τ (p), (τ, p) ∈ T × P. u∈Ω1 3 In practice, sub-patches near the border of M which have values outside of the domain Ω are discarded from the sum (2). 4 with for all u ∈ Ω1 , g1 (u; M ) := K2 (qy , qy0 0 ) = kqy kkqy0 0 k× X e−ku−zk 2 /2β12 exp(−kq̃y − q̃y0 0 k2 /2α22 ) h1 (z; M ), z∈Ω and, for all z ∈ Ω, h1 (z; M ) := kpz k y y h√ ηj e −kwj −p̃z k2 /α21 in1 j=1 , CKN-grad is fully invariant to color. It is the gradient along each spatial dimension with 1 × 1 sub-patches – that is, the sub-patch p̃z for this first layer is simply twodimensional and can be written pz = (Gx , Gy ). Because the features are normalized, the inner part of the match kernel kp̃z − p̃0z0 k is directly linked to the cosine of the angle between the two gradients, see [5, 27]. Indeed, an explicit approximation of the kernel K1 with n1 evenly distributed orientations θj = 2jπ/n1 , j ∈ {1, . . . , n1 } writes: ωj = 2wj /α12 and, considering a sample of n pairs of sub-patches {(pi , p0i )}i=1,...,n , we solve: 2  n1 n kwj −p̃i k2 kwj −p̃0i k2 kp̃i −p̃0i k2 X X − − − 2α2 α2 α2  e 1 1 1 − ηj e e min i=1 0 z z0 Figure 3. The two-layer convolutional kernel architecture. Each layer is a weighted match kernel between all patches of the previous one; qz (resp qz0 0 ) is a sub-patch of pz (resp p0z0 ), which is a itself a sub-patch of M (resp M 0 ). The two-layer CKN provides an approximate explicit feature map of this kernel. See [27] for details. bj = log(ηj )/2 − (1 + kwj k2 )/α12 wj ,ηj sub-patch qy0 0 sub-patch qy sub-patch p0z0 sub-patch pz z z where Ω1 is a subset of Ω as in [27] and w and η are learned parameters. There are two distinct approximations, one in the subsampling defined by |Ω1 | ≤ |Ω| that corresponds to the stride of a CNN pooling operation, and one in the embedding of the Gaussian kernel of the subpatches: k1 (pz , p0z0 ) ≈ h1 (z; M )h1 (z 0 ; M 0 ). Since K1 (M, M 0 ) is a sum of the match-kernel terms, we can approximate it at sub-patch level by solving an optimization problem. In contrast to the original formulation in Eq. 4 of [27], we introduce the change of variables 0 y y0 0 e−kp̃z −p̃z0 k 2 /2α21 j=1 We use stochastic gradient optimization to find a stationary point of this (non-convex) objective. This is much faster than the original L-BFGS optimizer [27]; see Sec. 6. Multi-layer CKN kernel. A kernel can be overlaid on top of the single kernel for a “deeper” and potentially better feature representation [4]. Given an input patch M , the single-layer CKN defines an approximation f1 (M ) that can be interpreted as a spatial map. It is possible to define a kernel K2 on this map in the same way as we have done for input patches. For that, we simply define a patch size, new hyper-parameters β2 and α2 , and replace M, M 0 by f1 (M ), f1 (M 0 ) in all equations of the previous section. Figure 3 gives an illustration of the corresponding two-layer convolutional kernel. Training a multi-layer CKN is naturally sequential, one layer after another. Input types. We investigate three possible inputs for our CKNs. The first, CKN-raw, directly feeds the raw RGB patch to the network. This scheme captures the hue information, which can prove a drawback in certain situations. CKN-white consists in pre-processing each sub-patch of the CKN’s first layer, by subtracting their mean color, and using PCA-whitening, with a PCA learned on all subpatches of the initial patch. This responds only to local variations inside the sub-patch, and makes the network more invariant to color. ≈ n1 X ϕ1 (j; pz )ϕ1 (j; p0z0 ), j=1 where for all j, 2 2 2 ϕ1 (j, pz ) = e−((cos θj −Gx /ρ) +(sin θj −Gy /ρ) )/α1 p and ρ = Gx + Gy . This formulation can be interpreted as a soft-binning of gradient orientations in a “histogram” of size n1 . To ensure an adequate distribution in each bin, 2
1/2 we set α1 = (1 − cos (2π/n1 ))2 + sin (2π/n1 ) . 5. Datasets We conduct experiments for two tasks, patch and image retrieval. We introduce a new dataset for both, which we describe in this section, together with the standard benchmarks. 5.1. Patch retrieval The Mikolajczyk Dataset. Designed to benchmark interest points detectors and descriptors, the Mikolajczyk dataset [29] contains a set of 8 scenes with 6 images for each. Images of a scene are linked by a homography. We extract regions with the Hessian-Affine detector, and match the corresponding descriptors using Euclidean nearest neighbor. The match between a pair of ellipses is counted correct if the projection of the region with the 5 ground-truth homography to the second image overlaps by at least 50%. Mean average precision (mAP) is used as performance measure. RomePatches. Since the existing patch retrieval datasets we are aware of do not contain color information and are not extracted with our detector (Hessian-Affine), we introduce a new dataset4 . Similar to [46], we use the 3D-reconstruction of landmarks to get different views of the same location. We use the Rome16K dataset [24], which consists of 16,179 images of locations in Rome, downloaded from photo sharing sites. Images are partitioned in 66 “bundles”, each one containing a set of viewpoints of a given location in Rome (e.g. “Trevi Fountain”). Within a bundle, consistent camera parameters are available for all images5 . We match the SIFT descriptors of all images using product quantization [17]. Then we keep only matches that verify the epipolar constraint within a tolerance of 3 pixels. Pairwise point matches are then aggregated greedily to form larger groups of 2D points viewed from several cameras. Groups are merged while the reproduction error from the estimated 3D position is below the 3 pixel threshold. Fig. 4 shows matching patches extracted with this algorithm. We split the dataset into two sets of bundles, the train set with 44 bundles on which we are allowed to learn parameters and tune hyperparameters. The remaining 22 bundles form the test set. From the train as well as the test set, we select 1,000 3D points that are viewed in at least 10 different images and use one as a query and nine randomly sampled as the targets. Our dataset therefore contains 9,000 target points, and 1,000 queries for the train as well as the test set, i.e., a total of 20,000 patches. We report mean average precision (mAP). Figure 4. Patch and image retrieval on the Rome dataset. Top: examples of matching patches. Bottom: Images of the same bundle, that therefore share the same class for image retrieval. of scenes and objects. 500 images are used as queries and the queries are excluded from the datasets. The standard metrics are mAP for Oxford, Paris and Holidays and 4×recall@4 for UKB. 6. Experimental Results After describing implementation details, we report results for patch and image retrieval. 6.1. Implementation details As our goal is to optimize local descriptors, all methods are given the same patch information as input (computed at Hessian-affine interest points), and are evaluated with the same global descriptor (VLAD with 256 centroids). We believe that improvements in feature detection and aggregation would benefit all architectures equally, without changing the relative performance of patch descriptors. 5.2. Image Retrieval RomePatches-Image. Using the aforementioned bundle split, we select 1,000 query images and 1,000 target images evenly distributed over all bundles for both train and test splits. Two images are considered to match if they come from the same bundle, as illustrated in Fig. 4. Patch extraction. As input for all methods, we use 51 × 51 pixel patches, which was found to be optimal on SIFT descriptors for the Oxford dataset. CNN implementation. For CNNs, we use the popular Caffe framework [20], and the provided AlexNet (learned on ImageNet 2012). For the PhilippNet [12], we used the model provided by the authors. As explained in section 4, we rescale the 51x51 input patches to the size that, when fed to the CNN, produces 1x1 output maps. Rescaling artifacts do not have a noticeable impact compared to re-extracting patches. Oxford. The Oxford dataset [35] involves 5,000 images of Oxford landmarks. 11 locations in the city are selected as queries. Each location is represented by 5 bounding boxes each extracted from a different image. Given one of the 55 bounding boxes, the task is to find all images of the same location. UKbench and Holidays. The University of Kentucky benchmark is a set of 10,200 photos. Each group of 4 images represents the same object. Each image is used as a query in turn. The Holidays dataset contains 1,491 photos Details of CKN learning. AlexNet and PhilippNet are provided with their parameters, we only learn CKNs. To do so, we randomly select a set of 100K patches in the train split of RomePatches. For each layer, 1 million sub-patches corresponding to convolution areas are extracted and all pairs of patches are fed to the objective function (4.2). The SGD optimization is run for 300K iterations with a batchsize of 4 Available online at http://lear.inrialpes.fr/people/ paulin/projects/RomePatches/ 5 http://www.cs.cornell.edu/projects/p2f/ 6 CKN-grad 1000. Because the objective is nonconvex, several tricks were used, such as random initialization, preconditioning (optimization is conducted in a space where the patch entries are decorrelated), selecting an initial learning rate in the range {1, 2−1/2 , 2−1 , . . . , 2−20 } by performing 1K iterations and choosing the one giving the lowest objective evaluated on a validation set [6]; after choosing the learning rate, we keep monitoring the objective on a validation set every 1K iteration, and perform backtracking√in case of divergence. The learning rate is also divided by 2 every 50K iterations. These heuristics are fixed over all experiments. Training a CKN takes roughly 10 min on a GPU compared to 2-3 days for the L-BFGS implementation of [27]. As CKN and CNN share the same architecture, the descriptor extraction time is similar for all convolutional methods. Layer 2 —2x2, 2, 512 4x4,2,1024 93 80 92 70 91 91 60 90 50 64 256 1024 90 64 no PCA PCA 256 1024 64 256 1024 PCA+whitening PCA+semi-whitening Figure 5. Influence of dimensionality reduction on patch retrieval performance. Results reported in mAP (%) on the train split of RomePatches as a function of the PCA dimension. As a comparison, SIFT reports 91.6%. Architecture RomePatches Miko. train test SIFT 51x51 128 91.6 87.9 57.8 AlexNet-conv1 11x11 96 66.4 65.0 40.9 AlexNet-conv2 51x51 256 73.8 69.9 46.4 AlexNet-conv3 99x99 384 81.6 79.2 53.7 AlexNet-conv4 131x131 384 78.4 75.7 43.4 AlexNet-conv5 163x163 256 53.9 49.6 24.4 PhilippNet 64x64 512 86.1 81.4 59.7 CKN-grad 51x51 1024 92.5 88.1 59.5 CKN-raw 51x51 1024 79.3 76.3 50.9 CKN-white 51x51 1024 91.9 87.7 62.5 Table 3. Results of convolutional architectures for patch retrieval. Because the evaluation is computationally cheaper for the patch retrieval task than for image retrieval (10K patches to encode for RomePatches, against more than 4M for Holidays), we optimize the hyperparameters of our CKNs on the RomePatches dataset. We select the best parameters on the train split, without accessing the test data. Parametric exploration of CKNs. We explore the three input types separately. For each layer, four hyperparameters have to be determined: the size of the convolutional mask (sub-patch size), the coefficient αk , the pooling factor and the number of outputs (nk ). The spatial comparison coefficient βk is related to the pooling factor and √ is set as in [27]—that is, to the pooling factor divided by 2. We determine αk as a quantile σk of the distribution of pairwise distances between sub-patches. This value was found optimal at 10−3 for all architectures, a much smaller value than reported in [27], which suggests that image classification requires more invariance than patch matching. As mentioned before, we optimize these parameters over the train split of RomePatches. We try the values 2, 3, 4 and 5 for the sub-patch sizes and pooling factors with 128, 256, 512 or 1024 outputs. The α parameter was selected in the {0.1, 0.01, 0.001} quantiles. The retained parameters are given in Table 2. To the notable exception of color, architectures perform better with two layers. In general, the higher the number of features, the better performance. Layer 1 5x5, 5, 512 3x3, 3, 512 1x1, 3, 16 CKN-white 90 92 6.2. Patch retrieval Input CKN-raw CKN-white CKN-grad CKN-raw 93 coverage Dim Table 2 for each input type. Comparative results. We compare the convolutional architectures on our three patch datasets: RomePatches-train, RomePatches-test and Mikolajczyk. Results are given in Table 3. For AlexNet CNNs, we report results for all outputs of the 5 convolutional layers (after ReLU). We note that SIFT is an excellent baseline for these methods, and that CNN architectures that were designed for local invariances perform better than the ones used in AlexNet, as observed in [12]. The results of the PhilippNet on the Mikolajczyk dataset are different from the ones reported in [12], for several reasons. First, we evaluate on Hessian-Affine descriptors while they use MSER. To have a comparable setting, we use their network with an input of 64x64, while they slide it on 91x91 patches. Such an additional layer results in a small increase of performance (2% for patch retrieval and 1% for image retrieval). We observe that PhilippNet outperforms both SIFT and AlexNet, which was the conclusion of [12]; CKN trained on whitened patches do however yield better results. dim. 41472 32768 50176 6.3. Image Retrieval Table 2. For each layer we indicate the sub-patch size, the subsampling factor and the number of filters. For the gradient network, the value 16 corresponds to the number of orientations. Settings. We learn a vocabulary of 256 centroids on a related database: for Holidays and UKB we use 5000 Flickr images and for Oxford, we train on Paris [36]. For In the following, we use the best architectures given in 7 SIFT AlexNet-conv1 AlexNet-conv2 AlexNet-conv3 AlexNet-conv4 AlexNet-conv5 PhilippNet CKN-grad CKN-raw CKN-white CKN-mix Holidays UKB Oxford 64.0 59.0 62.7 79.3 77.1 75.3 74.1 66.5 69.9 78.7 79.3 3.44 3.33 3.19 3.74 3.73 3.69 3.66 3.42 3.54 3.74 3.76 43.7 18.8 12.5 33.3 34.3 33.4 38.3 49.8 23.0 41.8 43.4 Method \ Dataset VLAD [19] VLAD++ [2] Global-CNN [3] MOP-CNN [14] Ours Rome train test 52.9 62.7 28.9 36.8 36.1 21.0 47.1 54.7 47.9 55.4 45.7 53.1 50.2 60.4 57.0 66.2 33.0 43.8 51.9 62.4 54.5 65.3 Holidays 63.4 64.6 79.3 80.2 79.3 UKB 3.47 3.56 3.76 Oxford 55.5* 54.5 49.8 (56.5*) Table 5. Comparison with state-of-the-art image retrieval results. Results with * use a Hessian-Affine detector with gravity assumption [32]. mark datasets and on the proposed RomePatches benchmark dataset. Acknowledgements. This work was partially supported by projects “Allegro” (ERC), “Titan” (CNRS-Mastodons), “Macaron” (ANR-14-CE23-0003-01), the Moore-Sloan Data Science Environment at NYU and a Xerox Research Center Europe collaboration contract. Table 4. Image retrieval results. CKN-mix is the result of the concatenation of the VLAD descriptors for the three channels. RomePatches-Train and RomePatches-Test the vocabulary is learned the other one. The final VLAD descriptor size is 256 times the local descriptor dimension. References Comparative results. We compare all convolutional approaches as well as the SIFT baseline in the image retrieval settings. Results are summarized in Table 4. On datasets for which color is dominant (e.g. Holidays or UKB), the best individual CKN results are attained by CKN-white, improved by combining the three channels. On images of buildings, gradients still perform best and the addition of color channels is harmful, which also explains the poor performance of AlexNet. On the other hand, PhilippNet was trained to be invariant to colorimetric transformations, and therefore yields better results than its CNN counterpart. [1] S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski. Building Rome in a day. Communications of the ACM, 2011. 2 [2] R. Arandjelovic and A. Zisserman. All about VLAD. In CVPR, 2013. 8 [3] A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky. Neural codes for image retrieval. In ECCV, 2014. 1, 2, 3, 8 [4] L. Bo, K. Lai, X. Ren, and D. Fox. Object recognition with hierarchical kernel descriptors. In CVPR, 2011. 5 [5] L. Bo, X. Ren, and D. Fox. Kernel descriptors for visual recognition. In NIPS, 2010. 2, 5 [6] L. Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade. Springer, 2012. 7 [7] M. Brown, G. Hua, and S. Winder. Discriminative learning of local image descriptors. PAMI, 2011. 2 [8] M. Calonder, V. Lepetit, C. Strecha, and P. Fua. BRIEF: Binary robust independent elementary features. In ECCV, 2010. 2 [9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 3 [10] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014. 1 [11] A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox. Discriminative unsupervised feature learning with convolutional neural networks. NIPS, 2014. 2, 4 [12] P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor matching with convolutional neural networks: a comparison to SIFT. arXiv Preprint, 2014. 1, 3, 4, 6, 7 [13] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 1, 2 Comparison with the state of the art. Table 5 compares our approach to recently published results. Approaches based on VLAD with SIFT [2, 19] can be improved significantly by CKN local descriptors (+15% on Holidays). To compare to the state of the art with SIFT on Oxford [2], we use the same Hessian-Affine patches extracted with gravity assumption [32]. Note that this alone results in a 7% gain. We also compare with global CNNs [3]. Our approach outperforms it on Oxford and UKB and is on par on Holidays. On Holidays, our approach is slightly below the one of [14], that uses AlexNet descriptors and VLAD pooling on large, densely extracted patches. Note that they perform dimensionality reduction and whitening, which results in a 2% improvement. We plan to investigate dimensionality reduction methods [3, 15] as well as quantization [17] in future work. 7. Conclusion We propose a new descriptor Patch-CKN for patch and image retrieval, that performs on par or better than supervised CNNs on standard patch and image retrieval bench8 [35] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. In CVPR, 2007. 6 [36] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Lost in quantization: Improving particular object retrieval in large scale image databases. In CVPR, 2008. 8 [37] J. Philbin, M. Isard, J. Sivic, and A. Zisserman. Descriptor learning for efficient retrieval. In ECCV. 2010. 2 [38] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. arXiv Preprint, 2014. 3 [39] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, and F. Moreno-Noguer. Fracking deep convolutional image descriptors. Arxiv preprint, 2015. 1, 3 [40] K. Simonyan, A. Vedaldi, and A. Zisserman. Learning local feature descriptors using convex optimisation. PAMI, 2014. 2 [41] J. Sivic and A. Zisserman. Video google: A text retrieval approach to object matching in videos. In ICCV, 2003. 1 [42] E. Tola, V. Lepetit, and P. Fua. Daisy: An efficient dense descriptor applied to wide-baseline stereo. PAMI, 2010. 2 [43] T. Tuytelaars and K. Mikolajczyk. Local invariant feature detectors: A survey. Foundations and Trends in Computer Graphics and Vision, 2008. 1 [44] A. Vedaldi and A. Zisserman. Efficient additive kernels via explicit feature maps. TPAMI, 2012. 2, 4 [45] Z. Wang, B. Fan, and F. Wu. Local intensity order pattern for feature description. In ICCV, 2011. 2 [46] S. Winder, G. Hua, and M. Brown. Picking the best daisy. In CVPR, 2009. 2, 3, 6 [47] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014. 2 [14] Y. Gong, L. Wang, R. Guo, and S. Lazebnik. Multi-scale orderless pooling of deep convolutional activation features. In ECCV, 2014. 3, 8 [15] H. Jégou and O. Chum. Negative evidences and cooccurrences in image retrieval: the benefit of PCA and whitening. In ECCV, 2012. 8 [16] H. Jegou, M. Douze, and C. Schmid. Hamming embedding and weak geometric consistency for large scale image search. In ECCV. 2008. 2 [17] H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. PAMI, 2011. 6, 8 [18] H. Jégou, M. Douze, C. Schmid, and P. Pérez. Aggregating local descriptors into a compact image representation. In CVPR, 2010. 3 [19] H. Jégou, F. Perronnin, M. Douze, J. Sánchez, P. Pérez, and C. Schmid. Aggregating local image descriptors into compact codes. PAMI, 2012. 1, 8 [20] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. 2014. 6 [21] J. Jiang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu. Learning fine-grained image similarity with deep ranking. In CVPR, 2014. 3 [22] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012. 1, 3, 4 [23] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Handwritten digit recognition with a back-propagation network. NIPS, 1989. 1 [24] Y. Li, N. Snavely, and D. P. Huttenlocher. Location recognition using prioritized feature matching. In ECCV. 2010. 6 [25] J. Long, N. Zhang, and T. Darrell. Do Convnets learn correspondances? In NIPS, 2014. 1, 2, 3 [26] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004. 1, 2, 3 [27] J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid. Convolutional kernel networks. In NIPS, 2014. 2, 3, 4, 5, 7 [28] K. Mikolajczyk and C. Schmid. Scale & affine invariant interest point detectors. IJCV, 2004. 3 [29] K. Mikolajczyk and C. Schmid. A performance evaluation of local descriptors. PAMI, 2005. 2, 5 [30] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. Van Gool. A comparison of affine region detectors. IJCV, 2005. 1, 3 [31] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In CVPR, 2014. 1, 2 [32] M. Perdoch, O. Chum, and J. Matas. Efficient representation of local geometry for large scale object retrieval. In CVPR, 2009. 8 [33] F. Perronnin and C. Dance. Fisher kernels on visual vocabularies for image categorization. In CVPR, 2007. 2 [34] F. Perronnin, J. Sánchez, and Y. Liu. Large-scale image categorization with explicit data embedding. In CVPR, 2010. 2, 4 9 FlowNet: Learning Optical Flow with Convolutional Networks Alexey Dosovitskiy∗, Philipp Fischer† ,∗ Eddy Ilg∗, Philip Häusser, Caner Hazırbaş, Vladimir Golkov† University of Freiburg Technical University of Munich {fischer,dosovits,ilg}@cs.uni-freiburg.de, {haeusser,hazirbas,golkov}@cs.tum.edu Patrick van der Smagt Technical University of Munich Daniel Cremers Technical University of Munich Thomas Brox University of Freiburg smagt@brml.org cremers@tum.de brox@cs.uni-freiburg.de Abstract Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks CNNs succeeded at. In this paper we construct CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a large synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps. Figure 1. We present neural networks which learn to estimate optical flow, being trained end-to-end. The information is first spatially compressed in a contractive part of the network and then refined in an expanding part. 1. Introduction Since it was not clear whether this task could be solved with a standard CNN architecture, we additionally developed an architecture with a correlation layer that explicitly provides matching capabilities. This architecture is trained end-to-end. The idea is to exploit the ability of convolutional networks to learn strong features at multiple levels of scale and abstraction and to help it with finding the actual correspondences based on these features. The layers on top of the correlation layer learn how to predict flow from these matches. Surprisingly, helping the network this way is not necessary and even the raw network can learn to predict optical flow with competitive accuracy. Convolutional neural networks have become the method of choice in many fields of computer vision. They are classically applied to classification [25, 24], but recently presented architectures also allow for per-pixel predictions like semantic segmentation [28] or depth estimation from single images [10]. In this paper, we propose training CNNs endto-end to learn predicting the optical flow field from a pair of images. While optical flow estimation needs precise per-pixel localization, it also requires finding correspondences between two input images. This involves not only learning image feature representations, but also learning to match them at different locations in the two images. In this respect, optical flow estimation fundamentally differs from previous applications of CNNs. ∗ These Training a network to predict generic optical flow requires a sufficiently large training set. Although data augmentation does help, the existing optical flow datasets are still too small to train a network on par with state of the art. Getting optical flow ground truth for realistic video material is known to be extremely difficult [7]. Trading in realism authors contributed equally by the Deutsche Telekom Stiftung † Supported 1 for quantity, we generate a synthetic Flying Chairs dataset which consists of random background images from Flickr on which we overlay segmented images of chairs from [1]. These data have little in common with the real world, but we can generate arbitrary amounts of samples with custom properties. CNNs trained on just these data generalize surprisingly well to realistic datasets, even without fine-tuning. Leveraging an efficient GPU implementation of CNNs, our method is faster than most competitors. Our networks predict optical flow at up to 10 image pairs per second on the full resolution of the Sintel dataset, achieving state-ofthe-art accuracy among real-time methods. 2. Related Work Optical Flow. Variational approaches have dominated optical flow estimation since the work of Horn and Schunck [19]. Many improvements have been introduced [29, 5, 34]. The recent focus was on large displacements, and combinatorial matching has been integrated into the variational approach [6, 35]. The work of [35] termed DeepMatching and DeepFlow is related to our work in that feature information is aggregated from fine to coarse using sparse convolutions and max-pooling. However, it does not perform any learning and all parameters are set manually. The successive work of [30] termed EpicFlow has put even more emphasis on the quality of sparse matching as the matches from [35] are merely interpolated to dense flow fields while respecting image boundaries. We only use a variational approach for optional refinement of the flow field predicted by the convolutional net and do not require any handcrafted methods for aggregation, matching and interpolation. Several authors have applied machine learning techniques to optical flow before. Sun et al. [32] study statistics of optical flow and learn regularizers using Gaussian scale mixtures; Rosenbaum et al. [31] model local statistics of optical flow with Gaussian mixture models. Black et al. [4] compute principal components of a training set of flow fields. To predict optical flow they then estimate coefficients of a linear combination of these ’basis flows’. Other methods train classifiers to select among different inertial estimates [21] or to obtain occlusion probabilities [27]. There has been work on unsupervised learning of disparity or motion between frames of videos using neural network models. These methods typically use multiplicative interactions to model relations between a pair of images. Disparities and optical flow can then be inferred from the latent variables. Taylor et al. [33] approach the task with factored gated restricted Boltzmann machines. Konda and Memisevic [23] use a special autoencoder called ‘synchrony autoencoder’. While these approaches work well in a controlled setup and learn features useful for activity recognition in videos, they are not competitive with classi- cal methods on realistic videos. Convolutional Networks. Convolutional neural networks trained with backpropagation [25] have recently been shown to perform well on large-scale image classification by Krizhevsky et al. [24]. This gave the beginning to a surge of works on applying CNNs to various computer vision tasks. While there has been no work on estimating optical flow with CNNs, there has been research on matching with neural networks. Fischer et al. [12] extract feature representations from CNNs trained in supervised or unsupervised manner and match these features based on Euclidean distance. Zbontar and LeCun [36] train a CNN with a Siamese architecture to predict similarity of image patches. A drastic difference of these methods to our approach is that they are patch based and leave the spatial aggregation to postprocessing, whereas the networks in this paper directly predict complete flow fields. Recent applications of CNNs include semantic segmentation [11, 15, 17, 28], depth prediction [10], keypoint prediction [17] and edge detection [13]. These tasks are similar to optical flow estimation in that they involve per-pixel predictions. Since our architectures are largely inspired by the recent progress in these per-pixel prediction tasks, we briefly review different approaches. The simplest solution is to apply a conventional CNN in a ‘sliding window’ fashion, hence computing a single prediction (e.g. class label) for each input image patch [8, 11]. This works well in many situations, but has drawbacks: high computational costs (even with optimized implementations involving re-usage of intermediate feature maps) and per-patch nature, disallowing to account for global output properties, for example sharp edges. Another simple approach [17] is to upsample all feature maps to the desired full resolution and stack them together, resulting in a concatenated per-pixel feature vector that can be used to predict the value of interest. Eigen et al. [10] refine a coarse depth map by training an additional network which gets as inputs the coarse prediction and the input image. Long et al. [28] and Dosovitskiy et al. [9] iteratively refine the coarse feature maps with the use of ‘upconvolutional’ layers 1 . Our approach integrates ideas from both works. Unlike Long et al., we ‘upconvolve’ not just the coarse prediction, but the whole coarse feature maps, allowing to transfer more high-level information to the fine prediction. Unlike Dosovitskiy et al., we concatenate the ‘upconvolution’ results with the features from the ‘contractive’ part of the network. 1 These layers are often named ’deconvolutional’, although the operation they perform is technically convolution, not deconvolution Figure 2. The two network architectures: FlowNetSimple (top) and FlowNetCorr (bottom). The green funnel is a placeholder for the expanding refinement part shown in Fig 3. The networks including the refinement part are trained end-to-end. panding parts are trained as a whole using backpropagation. Architectures we use are depicted in Figures 2 and 3. We now describe the two parts of networks in more detail. Figure 3. Refinement of the coarse feature maps to the high resolution prediction. 3. Network Architectures Convolutional neural networks are known to be very good at learning input–output relations given enough labeled data. We therefore take an end-to-end learning approach to predicting optical flow: given a dataset consisting of image pairs and ground truth flows, we train a network to predict the x–y flow fields directly from the images. But what is a good architecture for this purpose? Pooling in CNNs is necessary to make network training computationally feasible and, more fundamentally, to allow aggregation of information over large areas of the input images. But pooling results in reduced resolution, so in order to provide dense per-pixel predictions we need to refine the coarse pooled representation. To this end our networks contain an expanding part which intelligently refines the flow to high resolution. Networks consisting of contracting and ex- Contracting part. A simple choice is to stack both input images together and feed them through a rather generic network, allowing the network to decide itself how to process the image pair to extract the motion information. This is illustrated in Fig. 2 (top). We call this architecture consisting only of convolutional layers ‘FlowNetSimple’. Another approach is to create two separate, yet identical processing streams for the two images and to combine them at a later stage as shown in Fig. 2 (bottom). With this architecture the network is constrained to first produce meaningful representations of the two images separately and then combine them on a higher level. This roughly resembles the standard matching approach when one first extracts features from patches of both images and then compares those feature vectors. However, given feature representations of two images, how would the network find correspondences? To aid the network in this matching process, we introduce a ‘correlation layer’ that performs multiplicative patch comparisons between two feature maps. An illustration of the network architecture ‘FlowNetCorr’ containing this layer is shown in Fig. 2 (bottom). Given two multi-channel feature maps f1 , f2 : R2 → Rc , with w, h, and c being their width, height and number of channels, our correlation layer lets the network compare each patch from f1 with each path from f2 . For now we consider only a single comparison of two patches. The ’correlation’ of two patches centered at x1 in the first map and x2 in the second map is then defined as X c(x1 , x2 ) = hf1 (x1 + o), f2 (x2 + o)i (1) o∈[−k,k]×[−k,k] for a square patch of size K := 2k + 1. Note that Eq. 1 is identical to one step of a convolution in neural networks, but instead of convolving data with a filter, it convolves data with other data. For this reason, it has no trainable weights. Computing c(x1 , x2 ) involves c · K 2 multiplications. Comparing all patch combinations involves w2 · h2 such computations, yields a large result and makes efficient forward and backward passes intractable. Thus, for computational reasons we limit the maximum displacement for comparisons and also introduce striding in both feature maps. Given a maximum displacement d, for each location x1 we compute correlations c(x1 , x2 ) only in a neighborhood of size D := 2d + 1, by limiting the range of x2 . We use strides s1 and s2 , to quantize x1 globally and to quantize x2 within the neighborhood centered around x1 . In theory, the result produced by the correlation is fourdimensional: for every combination of two 2D positions we obtain a correlation value, i.e. the scalar product of the two vectors which contain the values of the cropped patches respectively. In practice we organize the relative displacements in channels. This means we obtain an output of size (w × h × D2 ). For the backward pass we implemented the derivatives with respect to each bottom blob accordingly. Expanding part. The main ingredient of the expanding part are ‘upconvolutional’ layers, consisting of unpooling (extending the feature maps, as opposed to pooling) and a convolution. Such layers have been used previously [38, 37, 16, 28, 9]. To perform the refinement, we apply the ‘upconvolution’ to feature maps, and concatenate it with corresponding feature maps from the ’contractive’ part of the network and an upsampled coarser flow prediction (if available). This way we preserve both the high-level information passed from coarser feature maps and fine local information provided in lower layer feature maps. Each step increases the resolution twice. We repeat this 4 times, resulting in a predicted flow for which the resolution is still 4 times smaller than the input. Overall architecture is shown in Figure 3. We found that further refinement from this resolution does not significantly improve the results, compared to a computationally less expensive bilinear upsampling to full image resolution. Variational refinement. In an alternative scheme, at this very last stage instead of bilinear upsampling we use the Ground truth FlowNetS FlowNetS+v Figure 4. The effect of variational refinement. In case of small motions (first row) the predicted flow is changed dramatically. For larger motions (second row), big errors are not corrected, but the flow field is smoothed, resulting in lower EPE. variational approach from [6] without the matching term: we start at the 4 times downsampled resolution and then use the coarse to fine scheme with 20 iterations to bring the flow field to the full resolution. Finally, we run 5 more iterations at the full image resolution. We additionally compute image boundaries with the approach from [26] and respect the detected boundaries by replacing the smoothness coefficient by α = exp(−λb(x, y)κ ), where b(x, y) denotes the thin boundary strength resampled at the respective scale and between pixels. This upscaling method is more computationally expensive than simple bilinear upsampling, but adds the benefits of variational methods to obtain smooth and subpixel-accurate flow fields. In the following, we denote the results obtained by this variational refinement with a ‘+v’ suffix. An example of variational refinement can be seen in Fig. 4. 4. Training Data Unlike traditional approaches, neural networks require data with ground truth not only for optimizing several parameters, but to learn to perform the task from scratch. In general, obtaining such ground truth is hard, because true pixel correspondences for real world scenes cannot easily be determined. An overview of the available datasets is given in Table 1. 4.1. Existing Datasets The Middlebury dataset [2] contains only 8 image pairs for training, with ground truth flows generated using four different techniques. Displacements are very small, typi- Middlebury KITTI Sintel Flying Chairs Frame pairs 72 194 1,041 22,872 Frames with ground truth 8 194 1,041 22,872 Ground truth density per frame 100% v50% 100% 100% Table 1. Size of already available datasets and the proposed Flying Chair dataset. cally below 10 pixels. The KITTI dataset [14] is larger (194 training image pairs) and includes large displacements, but contains only a very special motion type. The ground truth is obtained from real world scenes by simultaneously recording the scenes with a camera and a 3D laser scanner. This assumes that the scene is rigid and that the motion stems from a moving observer. Moreover, motion of distant objects, such as the sky, cannot be captured, resulting in sparse optical flow ground truth. The MPI Sintel [7] dataset obtains ground truth from rendered artificial scenes with special attention to realistic image properties. Two versions are provided: the Final version contains motion blur and atmospheric effects, such as fog, while the Clean version does not include these effects. Sintel is the largest dataset available (1,041 training image pairs for each version) and provides dense ground truth for small and large displacement magnitudes. 4.2. Flying Chairs The Sintel dataset is still too small to train large CNNs. To provide enough training data, we create a simple synthetic dataset, which we name Flying Chairs, by applying affine transformations to images collected from Flickr and a publicly available set of renderings of 3D chair models [1]. We retrieve 964 images from Flickr2 with a resolution of 1, 024 × 768 from the categories ‘city’ (321), ‘landscape’ (129) and ‘mountain’ (514). We cut the images into 4 quadrants and use the resulting 512 × 384 image crops as background. As foreground objects we add images of multiple chairs from [1] to the background. From the original dataset we remove very similar chairs, resulting in 809 chair types and 62 views per chair available. Examples are shown in Figure 5. To generate motion, we randomly sample 2D affine transformation parameters for the background and the chairs. The chairs’ transformations are relative to the background transformation, which can be interpreted as both the camera and the objects moving. Using the transformation parameters we generate the second image, the ground truth optical flow and occlusion regions. All parameters for each image pair (number, types, sizes and initial positions of the chairs; transformation parameters) are randomly sampled. We adjust the random distributions of these parameters in such a way that the resulting displacement histogram is similar to the one from Sintel (details can be found in the supplementary material). Using this procedure, we generate a dataset with 22,872 image pairs and flow fields (we re-use each background image multiple times). Note that this size is chosen arbitrarily and could be larger in principle. 2 Non-commercial public license. We use the code framework by Hays and Efros [18] 4.3. Data Augmentation A widely used strategy to improve generalization of neural networks is data augmentation [24, 10]. Even though the Flying Chairs dataset is fairly large, we find that using augmentations is crucial to avoid overfitting. We perform augmentation online during network training. The augmentations we use include geometric transformations: translation, rotation and scaling, as well as additive Gaussian noise and changes in brightness, contrast, gamma, and color. To be reasonably quick, all these operations are processed on the GPU. Some examples of augmentation are given in Fig. 5. As we want to increase not only the variety of images but also the variety of flow fields, we apply the same strong geometric transformation to both images of a pair, but additionally a smaller relative transformation between the two images. Specifically we sample translation from a the range [−20%, 20%] of the image width for x and y; rotation from [−17◦ , 17◦ ]; scaling from [0.9, 2.0]. The Gaussian noise has a sigma uniformly sampled from [0, 0.04]; contrast is sampled within [−0.8, 0.4]; multiplicative color changes to the RGB channels per image from [0.5, 2]; gamma values from [0.7, 1.5] and additive brightness changes using Gaussian with a sigma of 0.2. 5. Experiments We report the results of our networks on the Sintel, KITTI and Middlebury datasets, as well as on our synthetic Flying Chairs dataset. We also experiment with fine-tuning of the networks on Sintel data and variational refinement of the predicted flow fields. Additionally, we report runtimes of our networks, in comparison to other methods. 5.1. Network and Training Details The exact architectures of the networks we train are shown in Fig. 2. Overall, we try to keep the architectures of different networks consistent: they have nine convolutional layers with stride of 2 (the simplest form of pooling) in six of them and a ReLU nonlinearity after each layer. We do not have any fully connected layers, which allows the networks to take images of arbitrary size as input. Convolutional filter sizes decrease towards deeper layers of networks: 7 × 7 for the first layer, 5 × 5 for the following two layers and 3 × 3 starting from the fourth layer. The number of feature maps increases in the deeper layers, roughly doubling after each layer with a stride of 2. For the correlation layer in FlowNetC we chose the parameters k = 0, d = 20, s1 = 1, s2 = 2. As training loss we use the endpoint error (EPE), which is the standard error measure for optical flow estimation. It is the Euclidean distance between the predicted flow vector and the ground truth, averaged over all pixels. Figure 5. Flying Chairs. Generated image pair and color coded flow field (first three columns), augmented image pair and corresponding color coded flow field respectively (last three columns). For training CNNs we use a modified version of the caffe [20] framework. We choose Adam [22] as optimization method because for our task it shows faster convergence than standard stochastic gradient descent with momentum. We fix the parameters of Adam as recommended in [22]: β1 = 0.9 and β2 = 0.999. Since, in a sense, every pixel is a training sample, we use fairly small mini-batches of 8 image pairs. We start with learning rate λ = 1e−4 and then divide it by 2 every 100k iterations after the first 300k. With FlowNetCorr we observe exploding gradients with λ = 1e−4. To tackle this problem, we start by training with a very low learning rate λ = 1e−6, slowly increase it to reach λ = 1e−4 after 10k iterations and then follow the schedule just described. To monitor overfitting during training and fine-tuning, we split the Flying Chairs dataset into 22, 232 training and 640 test samples and split the Sintel training set into 908 training and 133 validation pairs. We found that upscaling the input images during testing may improve the performance. Although the optimal scale depends on the specific dataset, we fixed the scale once for each network for all tasks. For FlowNetS we do not upscale, for FlowNetC we chose a factor of 1.25. Fine-tuning. The used datasets are very different in terms of object types and motions they include. A standard solution is to fine-tune the networks on the target datasets. The KITTI dataset is small and only has sparse flow ground truth. Therefore, we choose to fine-tune on the Sintel training set. We use images from the Clean and Final versions of Sintel together and fine-tune using a low learning rate λ = 1e−6 for several thousand iterations. For best performance, after defining the optimal number of iterations using a validation set, we then fine-tune on the whole training set for the same number of iterations. In tables we denote finetuned networks with a ‘+ft’ suffix. 5.2. Results Table 2 shows the endpoint error (EPE) of our networks and several well-performing methods on public datasets (Sintel, KITTI, Middlebury), as well as on our Flying Chairs dataset. Additionally we show runtimes of different methods on Sintel. The networks trained just on the non-realistic Flying Chairs perform very well on real optical flow datasets, beating for example the well-known LDOF [6] method. After fine-tuning on Sintel our networks can outperform the competing real-time method EPPM [3] on Sintel Final and KITTI while being twice as fast. Sintel. From Table 2 one can see that FlowNetC is better than FlowNetS on Sintel Clean, while on Sintel Final the situation changes. On this difficult dataset, FlowNetS+ft+v is even on par with DeepFlow. Since the average endpoint error often favors over-smoothed solutions, it is interesting to see qualitative results of our method. Figure 6 shows examples of the raw optical flow predicted by the two FlowNets (without fine-tuning), compared to ground truth and EpicFlow. The figure shows how the nets often produce visually appealing results, but are still worse in terms of endpoint error. Taking a closer look reveals that one reason for this may be the noisy non-smooth output of the nets especially in large smooth background regions. This we can partially compensate with variational refinement. KITTI. The KITTI dataset contains strong projective transformations which are very different from what the networks encountered during training on Flying Chairs. Still, the raw network output is already fairly good, and additional fine-tuning and variational refinement give a further boost. Interestingly, fine-tuning on Sintel improves the results on KITTI, probably because the images and motions in Sintel are more natural than in Flying Chairs. The FlowNetS outperforms FlowNetC on this dataset. Flying Chairs. Our networks are trained on the Flying Chairs, and hence are expected to perform best on those. When training, we leave aside a test set consisting of 640 images. Table 2 shows the results of various methods on this Method EpicFlow [30] DeepFlow [35] EPPM [3] LDOF [6] FlowNetS FlowNetS+v FlowNetS+ft FlowNetS+ft+v FlowNetC FlowNetC+v FlowNetC+ft FlowNetC+ft+v Sintel Clean train test 2.27 4.12 3.19 5.38 6.49 4.19 7.56 4.50 7.42 3.66 6.45 (3.66) 6.96 (2.97) 6.16 4.31 7.28 3.57 6.27 (3.78) 6.85 (3.20) 6.08 Sintel Final train test 3.57 6.29 4.40 7.21 8.38 6.28 9.12 5.45 8.43 4.76 7.67 (4.44) 7.76 (4.07) 7.22 5.87 8.81 5.25 8.01 (5.28) 8.51 (4.83) 7.88 KITTI train test 3.47 3.8 4.58 5.8 9.2 13.73 12.4 8.26 6.50 7.52 9.1 6.07 7.6 9.35 7.45 8.79 7.31 - Middlebury train AEE AAE 0.31 3.24 0.21 3.04 0.45 4.97 1.09 13.28 0.33 3.87 0.98 15.20 0.32 3.84 1.15 15.64 0.34 3.92 0.93 12.33 0.33 3.81 Middlebury test AEE AAE 0.39 3.55 0.42 4.22 0.33 3.36 0.56 4.55 0.47 4.58 0.50 4.52 Chairs test 2.94 3.53 3.47 2.71 2.86 3.04 3.03 2.19 2.61 2.27 2.67 Time (sec) CPU GPU 16 17 0.2 65 2.5 0.08 1.05 0.08 1.05 0.15 1.12 0.15 1.12 Table 2. Average endpoint errors (in pixels) of our networks compared to several well-performing methods on different datasets. The numbers in parentheses are the results of the networks on data they were trained on, and hence are not directly comparable to other results. Images Ground truth EpicFlow FlowNetS FlowNetC Figure 7. Examples of optical flow prediction on the Flying Chairs dataset. The images include fine details and small objects with large displacements which EpicFlow often fails to find. The networks are much more successful. test set, some example predictions are shown in Fig. 7. One can see that FlowNetC outperforms FlowNetS and that the nets outperform all state-of-the-art methods. Another interesting finding is that this is the only dataset where the variational refinement does not improve performance but makes things worse. Apparently the networks can do better than variational refinement already. This indicates that with a more realistic training set, the networks might also perform even better on other data. Timings. In Table 2 we show the per-frame runtimes of different methods in seconds. Unfortunately, many methods only provide the runtime on a single CPU, whereas our FlowNet uses layers only implemented on GPU. While the error rates of the networks are below the state of the art, they are the best among real-time methods. For both training and testing of the networks we use an NVIDIA GTX Titan GPU. The CPU timings of DeepFlow and EpicFlow are taken from [30], while the timing of LDOF was computed on a single 2.66GHz core. 5.3. Analysis Training data. To check if we benefit from using the Flying Chairs dataset instead of Sintel, we trained a network just on Sintel, leaving aside a validation set to control the performance. Thanks to aggressive data augmentation, even Sintel alone is enough to learn optical flow fairly well. When testing on Sintel, the network trained exclusively on Sintel has EPE roughly 1 pixel higher than the net trained on Flying Chairs and fine-tuned on Sintel. The Flying Chairs dataset is fairly large, so is data augmentation still necessary? The answer is positive: training a network without data augmentation on the Flying Chairs results in an EPE increase of roughly 2 pixels when testing on Sintel. Comparing the architectures. The results in Table 2 allow to draw conclusions about strengths and weaknesses of the two architectures we tested. First, FlowNetS generalizes to Sintel Final better than FlowNetC. On the other hand, FlowNetC outperforms FlowNetS on Flying chairs and Sintel Clean. Note that Flying Chairs do not include motion blur or fog, as in Sintel Final. These results together suggest that even though the number of parameters of the two networks is virtually the same, the FlowNetC slightly more overfits to the training data. This does not mean the network remembers the training samples by heart, but it adapts to the kind of data it is presented during training. Though in our current setup this can be seen as a weakness, if better training data were available it could become an advantage. Second, FlowNetC seems to have more problems with large displacements. This can be seen from the results on KITTI discussed above, and also from detailed performance analysis on Sintel Final (not shown in the tables). FlowNetS+ft achieves s40+ error (EPE on pixels with displacements of at least 40 pixels) of 43.3px, and for FlowNetC+ft this value is 48px. One explanation is that the maximum displacement of the correlation does not allow to predict very large motions. This range can be increased at the cost of computational efficiency. Images Ground truth EpicFlow FlowNetS FlowNetC Figure 6. Examples of optical flow prediction on the Sintel dataset. In each row left to right: overlaid image pair, ground truth flow and 3 predictions: EpicFlow, FlowNetS and FlowNetC. Endpoint error is shown for every frame. Note that even though the EPE of FlowNets is usually worse than that of EpicFlow, the networks often better preserve fine details. 6. Conclusion Building on recent progress in design of convolutional network architectures, we have shown that it is possible to train a network to directly predict optical flow from two input images. Intriguingly, the training data need not be realistic. The artificial Flying Chairs dataset including just affine motions of synthetic rigid objects is sufficient to predict optical flow in natural scenes with competitive accuracy. This proves the generalization capabilities of the presented networks. On the test set of the Flying Chairs the CNNs even outperform state-of-the-art methods like DeepFlow and EpicFlow. It will be interesting to see how future networks perform as more realistic training data becomes available. Acknowledgments The work was partially funded by the ERC Starting Grants VideoLearn and ConvexVision, by the DFG Grants BR-3815/7-1 and CR 250/13-1, and by the EC FP7 project 610967 (TACMAN). References [1] M. Aubry, D. Maturana, A. Efros, B. Russell, and J. Sivic. Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models. In CVPR, 2014. 2, 5 [2] S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. Technical Report MSR-TR-2009-179, December 2009. 4 [3] L. Bao, Q. Yang, and H. Jin. Fast edge-preserving patchmatch for large displacement optical flow. In CVPR, 2014. 6, 7 [4] M. J. Black, Y. Yacoob, A. D. Jepson, and D. J. Fleet. Learning parameterized models of image motion. In CVPR, 1997. 2 [5] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. In ECCV, volume 3024, pages 25–36. Springer, 2004. 2 [6] T. Brox and J. Malik. Large displacement optical flow: descriptor matching in variational motion estimation. PAMI, 33(3):500–513, 2011. 2, 4, 6, 7 [7] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor, ECCV, Part IV, LNCS 7577, pages 611–625. Springer-Verlag, Oct. 2012. 1, 5 [8] D. C. Ciresan, L. M. Gambardella, A. Giusti, and J. Schmidhuber. Deep neural networks segment neuronal membranes in electron microscopy images. In NIPS, pages 2852–2860, 2012. 2 [9] A. Dosovitskiy, J. T. Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. In CVPR, 2015. 2, 4 [10] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. NIPS, 2014. 1, 2, 5 [11] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. PAMI, 35(8):1915– 1929, 2013. 2 [12] P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor matching with convolutional neural networks: a comparison to SIFT. 2014. pre-print, arXiv:1405.5769v1 [cs.CV]. 2 [13] Y. Ganin and V. S. Lempitsky. Nˆ4 -fields: Neural network nearest neighbor fields for image transforms. In ACCV, pages 536–551, 2014. 2 [14] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013. 5 [15] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 2 [16] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014. 4 [17] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and fine-grained localization. CVPR, 2015. 2 [18] J. Hays and A. A. Efros. im2gps: estimating geographic information from a single image. In CVPR, 2008. 5 [19] B. K. P. Horn and B. G. Schunck. Determining optical flow. Artificial Intelligence, 17:185–203, 1981. 2 [20] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 6 [21] R. Kennedy and C. Taylor. Optical flow with geometric occlusion estimation and fusion of multiple frames. In EMMCVPR. 2015. 2 [22] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6 [23] K. R. Konda and R. Memisevic. Unsupervised learning of depth and motion. CoRR, abs/1312.3429, 2013. 2 [24] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106–1114, 2012. 1, 2, 5 [25] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989. 1, 2 [26] M. Leordeanu, R. Sukthankar, and C. Sminchisescu. Efficient closed-form solution to generalized boundary detection. In Proceedings of the 12th European Conference on Computer Vision - Volume Part IV, ECCV’12, pages 516– 529, Berlin, Heidelberg, 2012. Springer-Verlag. 4 [27] M. Leordeanu, A. Zanfir, and C. Sminchisescu. Locally affine sparse-to-dense matching for motion and occlusion estimation. ICCV, 0:1721–1728, 2013. 2 [28] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. 1, 2, 4 [29] E. Mémin and P. Pérez. Dense estimation and object-based segmentation of the optical flow with robust techniques. 7(5):703–719, May 1998. 2 [30] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid. EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow. In CVPR, Boston, United States, June 2015. 2, 7 [31] D. Rosenbaum, D. Zoran, and Y. Weiss. Learning the local statistics of optical flow. In NIPS, 2013. 2 [32] D. Sun, S. Roth, J. Lewis, and M. J. Black. Learning optical flow. In ECCV, 2008. 2 [33] G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler. Convolutional learning of spatio-temporal features. In ECCV, pages 140–153, 2010. 2 [34] A. Wedel, D. Cremers, T. Pock, and H. Bischof. Structureand motion-adaptive regularization for high accuracy optic flow. In ICCV, Kyoto, Japan, 2009. 2 [35] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. DeepFlow: Large displacement optical flow with deep matching. In ICCV, Sydney, Australia, Dec. 2013. 2, 7 [36] J. Zbontar and Y. LeCun. Computing the stereo matching cost with a convolutional neural network. CoRR, abs/1409.4326, 2014. 2 [37] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014. 4 [38] M. D. Zeiler, G. W. Taylor, and R. Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In ICCV, pages 2018–2025, 2011. 4 Active Object Localization with Deep Reinforcement Learning Juan C. Caicedo Fundación Universitaria Konrad Lorenz Bogotá, Colombia Svetlana Lazebnik University of Illinois at Urbana Champaign Urbana, IL, USA juanc.caicedor@konradlorenz.edu.co slazebni@illinois.edu Sequence of attended regions to localize the object Abstract States We present an active detection model for localizing objects in scenes. The model is class-specific and allows an agent to focus attention on candidate regions for identifying the correct location of a target object. This agent learns to deform a bounding box using simple transformation actions, with the goal of determining the most specific location of target objects following top-down reasoning. The proposed localization agent is trained using deep reinforcement learning, and evaluated on the Pascal VOC 2007 dataset. We show that agents guided by the proposed model are able to localize a single instance of an object after analyzing only between 11 and 25 regions in an image, and obtain the best detection results among systems that do not use object proposals for object localization. … … Actions Steps 𝑡1 … 𝑡𝑖 𝑡𝑖 + 1 … 𝑡𝑛 − 1 𝑡𝑛 Figure 1. A sequence of actions taken by the proposed algorithm to localize a cow. The algorithm attends regions and decides how to transform the bounding box to progressively localize the object. tures to achieve state-of-the-art results in the Pascal and ImageNet benchmarks. Several other works have proposed the use of high-capacity CNNs to directly predict bounding boxes under a regression setting also with good results [27, 28, 9]. In this work, we propose a class-specific active detection model that learns to localize target objects known by the system. The proposed model follows a top-down search strategy, which starts by analyzing the whole scene and then proceeds to narrow down the correct location of objects. This is achieved by applying a sequence of transformations to a box that initially covers a large region of the image and is finally reduced to a tight bounding box. The sequence of transformations is decided by an agent that analyzes the content of the currently visible region to select the next best action. Each transformation should keep the object inside the visible region while cutting off as much background as possible. Figure 1 illustrates some steps of the dynamic decision process to localize a cow in an image. The proposed approach is fundamentally different from most localization strategies. In contrast to sliding windows, our approach does not follow a fixed path to search objects; instead, different objects in different scenes will end up in different search paths. Unlike object proposal algorithms, candidate regions in our approach are selected by a highlevel reasoning strategy instead of following low-level cues. Also, compared to bounding box regression algorithms, our approach does not localize objects following a single, struc- 1. Introduction The process of localizing objects with bounding boxes can be seen as a control problem with a sequence of steps to refine the geometry of the box. Determining the exact location of a target object in a scene requires active engagement to understand the context, change the fixation point, identify distinctive parts that support recognition, and determine the correct proportions of the box. During the last decade, the problem of object detection or localization has been studied by the vision community with the goal of recognizing the category of an object, and identifying its spatial extent with a tight bounding box that covers all its visible parts [10, 26]. This is a challenging setup that requires computation and analysis in multiple image regions, and a good example of a task driven by active attention. Important progress for improving the accuracy of object detectors has been recently possible with Convolutional Neural Networks (CNNs), which leverage big visual data and deep learning for image categorization. A successful model is the R-CNN detector proposed by Girshick et al. [12, 25], which combines object proposals and CNN fea1 tured prediction method. We propose a dynamic attentionaction strategy that requires to pay attention to the contents of the current region, and to transform the box in such a way that the target object is progressively more focused. To stimulate the attention of the proposed agent, we use a reward function proportional to how well the current box covers the target object. We incorporate the reward function in a reinforcement learning setting to learn a localization policy, based on the DeepQNetwork algorithm [23]. As a result, the trained agent can localize a single instance of an object in about 11 steps, which means that the algorithm can correctly find an object after processing only 11 regions of the image. We conducted a comprehensive experimental evaluation in the challenging Pascal VOC dataset, obtaining competitive results in terms of precision and recall. In what follows, we present and discuss the components of the proposed approach and provide a detailed analysis of experimental results. 2. Previous Works Object localization has been successfully approached with sliding window classifiers. A popular sliding window method, based on HOG templates and SVM classifiers, has been extensively used to localize objects [11, 21], parts of objects [8, 20], discriminative patches [29, 17] and even salient components of scenes [24]. Sliding windows are related to our work because they are category-specific localization algorithms, which is also part of our design. However, unlike our work, sliding windows make an exhaustive search over the location-scale space. A recent trend for object localization is the generation of category independent object proposals. Hosang et al. [15] provide an in depth analysis of ten object proposal methods, whose goal is to generate the smallest set of candidate regions with the highest possible recall. Substantial acceleration is achieved by reducing the set of candidates in this way, compared to sliding windows. Nonetheless, object detection based on proposals follows the same design of window-based classification on a set of reduced regions, which is still large (thousands of windows) for a single image that may contain a few interesting objects. Several works attempt to reduce the number of evaluated regions during the detection process. For instance, Lampert et al. [18] proposed a branch-and-bound algorithm to find high-scoring regions only evaluating a few locations. Recently, Gonzalez-Garcia et al. [13] proposed an active search strategy to accelerate category-specific R-CNN detectors. These methods are related to ours because they try to optimize computational resources for localization. Also related is the work of Divvala et al. [7], which uses context to determine the localization of objects. Visual attention models have been investigated with the goal of predicting where an observer is likely to orient the gaze (see [3] for a recent survey). These models are generally based on a saliency map that aggregates low-level features to identify interesting regions. These models are designed to predict human fixations and evaluate performance with user studies [33], while our work aims to localize objects and we evaluate performance in this task. There is recent interest in attention capabilities for visual recognition in the machine learning community. Xu et al. [35] use a Recurrent Neural Network (RNN) to generate captions for images, using an attention mechanism that explains where the system focused attention to predict words. Mnih et al. [22] and Ba et al. [2] also used RNNs to select a sequence of regions that need more attention, which are processed at higher resolution for recognizing multiple characters. Interestingly, these models are trained with Reinforcement Learning as we do; however, our work uses a simpler architecture and intuitive actions to transform boxes. 3. Object Localization as a Dynamic Decision Process We cast the problem of object localization as a Markov decision process (MDP) since this setting provides a formal framework to model an agent that makes a sequence of decisions. Our formulation considers a single image as the environment, in which the agent transforms a bounding box using a set of actions. The goal of the agent is to land a tight box in a target object that can be observed in the environment. The agent also has a state representation with information of the currently visible region and past actions, and receives positive and negative rewards for each decision made during the training phase. During testing, the agent does not receive rewards and does not update the model either, it just follows the learned policy. Formally, the MDP has a set of actions A, a set of states S, and a reward function R. This section presents details of these three components, and the next section presents technical details of the training and testing strategies. 3.1. Localization Actions The set of actions A is composed of eight transformations that can be applied to the box and one action to terminate the search process. These actions are illustrated in Figure 2, and are organized in four sub-sets: actions to move the box in the horizontal and vertical axes, actions to change scale, and actions to modify aspect ratio. In this way, the agent has four degrees of freedom to transform the box during any interaction with the environment. A box is represented by the coordinates in pixels of its two corners: b = [x1 , y1 , x2 , y2 ]. Any of the transformation actions make a discrete change to the box by a factor relative to its current size in the following way: αw = α ∗ (x2 − x1 ) αh = α ∗ (y2 − y1 ) (1) Horizontal moves Right Left Vertical moves Up Down Scale changes Bigger Smaller Aspect ratio changes Fatter Taller Trigger Figure 2. Illustration of the actions in the proposed MDP, giving 4 degrees of freedom to the agent for transforming boxes. where α ∈ [0, 1]. Then, the transformations are obtained by adding or removing αw or αh to the x or y coordinates, depending on the desired effect. For instance, a horizontal move to the right adds αw to x1 and xy , while decreasing aspect ratio subtracts αw from x1 , and adds it to x2 . Note that the origin in the image plane is located in the top-left corner. We set α = 0.2 in all our experiments, since this value gives a good trade-off between speed and localization accuracy. In early exploration experiments we noticed that smaller values make the agent slower to localize objects, while larger values make it harder to place the box correctly. Finally, the only action that does not transform the box is a trigger to indicate that an object is correctly localized by the current box. This action terminates the sequence of the current search, and restarts the box in an initial position to begin the search for a new object. The trigger also modifies the environment: it marks the region covered by the box with a black cross as shown in the final frame of the top two examples in Figure 6. This mark serves as a inhibitionof-return (IoR) mechanism by which the currently attended region is prevented from being attended again. IoR mechanisms have been widely used in visual attention models (see [16] for a review) to suppress the attended location and avoid endless attractions towards the most salient stimulus. 3.2. State The state representation is a tuple (o, h), where o is a feature vector of the observed region, and h is a vector with the history of taken actions. The set of possible states S is very large as it includes arbitrary boxes from a large set of images, and is expanded with all the combinations of actions that lead to those boxes. Therefore, generalization is important to design an effective state representation. The feature vector o is extracted from the current region using a pre-trained CNN following the architecture of Zeiler and Fergus [36]. Any attended region by the agent is warped to match the input of the network (224 × 224) regardless of its size and aspect ratio, following the technique proposed by Girshick et al. [12]. We also expand the region to include 16 pixels of context around the original box. We forward the region up to the layer 6 (fc6) and use the 4,096 dimensional feature vector to represent its content. The history vector h is a binary vector that informs which actions have been used in the past. Each action in the history vector is represented by a 9-dimensional binary vector, where all values are zero except the one corresponding to the taken action. The history vector encodes 10 past actions, which means that h ∈ R90 . Although h is very lowdimensional compared to o, it has enough energy to inform what has happened in the past. This information demonstrated to be useful to stabilize search trajectories that might get stuck in repetitive cycles, improving average precision by approximately 3 percent points. The history vector also works better than appending a few more frames to the state representation with the additional benefit of increasing dimensionality by a negligible factor. 3.3. Reward Function The reward function R is proportional to the improvement that the agent makes to localize an object after selecting a particular action. Improvement in our setup is measured using the Intersection-over-Union (IoU) between the target object and the predicted box at any given time. More specifically, the reward function is estimated using the differential of IoU from one state to another. The reward function can be estimated during the training phase only because it requires ground truth boxes to be calculated. Let b be the box of an observable region, and g the ground truth box for a target object. Then, IoU between b and g is defined as IoU (b, g) = area(b ∩ g)/area(b ∪ g). The reward function Ra (s, s0 ) is granted to the agent when it chooses the action a to move from state s to s0 . Each state s has an associated box b that contains the attended region. Then, the reward is as follows1 : Ra (s, s0 ) = sign (IoU (b0 , g) − IoU (b, g)) (2) Intuitively, equation 2 says that the reward is positive if IoU improved from state s to state s0 , and negative otherwise. This reward scheme is binary r ∈ {−1, +1}, and applies to any action that transforms the box. Without quantization, the difference in IoU is small enough to confuse the agent about which actions are good or bad choices. Binary rewards communicate more clearly which transformations keep the object inside the box, and which ones take the box away the target. In this way, the agent pays a penalty for taking the box away the target, and is rewarded to keep the target object in the visible region until there is no other transformation that improves localization. In that case, the best action to choose should be the trigger. The trigger has a different reward scheme because it leads to a terminal state that does not change the box, and thus, the differential of IoU will always be zero for this action. The reward for the trigger is a thresholding function of IoU as follows: 1 Notice that the ground truth box g is part of the environment and cannot be modified by the agent. action history Rω (s, s0 ) =  +η −η if IoU (b, g) ≥ τ otherwise (3) where ω is the trigger action, η is the trigger reward, set to 3.0 in our experiments, and τ is a threshold that indicates the minimum IoU allowed to consider the attended region as a positive detection. The standard threshold for object detection evaluation is 0.5, but we used τ = 0.6 during training to encourage better localization. A larger value for τ has a negative effect in performance because the agent learns that only clearly visible objects are worth the trigger, and tends to avoid truncated or naturally occluded objects. Finally, the proposed reward scheme implicitly considers the number of steps as a cost because of the way in which Q-learning models the discount of future rewards (positive and negative). The agent follows a greedy strategy, which prefers short sequences because any unnecessary step pays a penalty that reduces the accumulated utility. 4. Finding a Localization Policy with Reinforcement Learning The goal of the agent is to transform a bounding box by selecting actions in a way that maximizes the sum of the rewards received during an interaction with the environment (an episode). The core problem is to find a policy that guides the decision making process of this agent. A policy is a function π(s) that specifies the action a to be chosen when the current state is s. Since we do not have the state transition probabilities and the reward function is data-dependent, the problem is formulated as a reinforcement learning problem using Q-learning [31]. In our work, we follow the deep Q-learning algorithm recently proposed by Mnih et al. [23]. This approach estimates the action-value function using a neural network, and has several advantages over previous Q-learning methods. First, the output of the Q-network has as many units as actions in the problem. This makes the model efficient because the input image is forwarded through the network only once to estimate the value of all possible actions. Second, the algorithm incorporates a replay-memory to collect various experiences and learns from them in the long run. In this way, transitions in the replay-memory are used in many model updates resulting in greater data efficiency. Third, to update the model the algorithm selects transitions from the replay-memory uniformly at random to break short-term correlations between states. This makes the algorithm more stable and prevents divergence of the parameters. After learning the action-value function Q(s, a), the policy that the agent follows is to select the action a with the maximum estimated value. Size: 224 pixels 5 conv layers 4096 units Layer 6 Input region 1024 units 1024 units 9 actions Layer 1 Layer 2 Output Pre-trained CNN Deep QNetwork Figure 3. Architecture of the proposed QNetwork. The input region is first warped to 224 × 224 pixels and processed by a pretrained CNN with 5 convolutional layers and 1 fully connected layer. The output of the CNN is concatenated with the action history vector to complete the state representation. It is processed by the Q-network which predicts the value of the 9 actions. 4.1. Q-learning for Object Localization We use a Q-network that takes as input the state representation discussed in section 3.2 and gives as output the value of the nine actions presented in section 3.1. We train category specific Q-networks following the architecture illustrated in Figure 3. Notice that in our design we do not learn the full feature hierarchy of the convolutional network; instead, we rely on a pre-trained CNN. Using a pre-trained CNN has two advantages: First, learning the Q function is faster because we need to update the parameters of the Q-Network only, while using the deep CNN just as a feed-forward feature extractor. Second, the hierarchy of features is trained with a larger dataset, leveraging generic discriminative features in our method. Learning the full hierarchy of features is possible under the deep Q-learning framework, and we hypothesize that performance could be improved because the learned features would be adapted for the localization task instead of classification. However, this mainly requires larger object detection datasets, so we leave this possibility for future work. 4.2. Training Localization Agents The parameters of the Q-network are initialized at random. Then, the agent is set to interact with the environment in multiple episodes, each presenting a different training image. The policy followed during training is -greedy [31], which gradually shifts from exploration to exploitation according to the value of . During exploration, the agent selects actions randomly to observe different transitions and collects a varied set of experiences. During exploitation, the agent selects actions greedily according to the learned policy, and learns from its own successes and mistakes. In our work, exploration does not proceed with random actions. Instead, we use a guided exploration strategy following the principles of apprenticeship learning [1, 6, 19], which is based on demonstrations made by an expert to the agent. Since the environment knows the ground truth boxes, and the reward function is calculated with respect to the IoU with the current box, we can identify which actions will give positive and negative rewards. Then, during exploration, the agent is allowed to choose one random action from the set of positive actions 2 . Notice that for one particular state s, there might be multiple positive actions because there is no single path to localize an object. Using this strategy, the algorithm terminates in a small number of epochs. The -greedy training strategy is run for 15 epochs, each completed after the agent has had interaction with all training images. During the first 5 epochs,  is annealed linearly from 1.0 to 0.1 to progressively let the agent use its own learned model. After the fifth epoch,  is fixed to 0.1 so the agent adjusts the model parameters from experiences produced by its own decisions. The parameters are updated using stochastic gradient descent and the back-propagation algorithm, and we also use dropout regularization [30]. 4.3. Testing a Localization Agent Once an agent is trained with the procedure described above, it learns to attend regions that contain objects of the target category. Since we do not know the number of objects present in a single image beforehand, we let the agent run for a maximum of 200 steps, so only 200 regions are evaluated per test image. An alternative to stop the search after a fixed number of steps is to include an extra termination action to let the agent indicate when the search is done. However, additional actions introduce new errors and make the problem more difficult to learn. We simplified the model with a minimum set of actions to act locally in time. At each step, the agent makes a decision to transform the current box or selects the trigger to indicate that an object has been found. When the trigger is used, the search for other objects continues from a new box that covers a large portion of the image. The search for objects is restarted from the beginning due to two possible events: the agent used the trigger, or 40 steps passed without using the trigger. We found that 40 steps are enough to localize most objects, even the smaller ones, and when it takes longer is usually because the agent is stuck searching in an ambiguous region. Restarting the box helps to take a new perspective of the scene. The very first box covers the entire scene, and after any restarting event, the new box has a reduced size set to 75% of the image size, and is placed in one of the four corners of the image always in the same order (from top-left to right-bottom). 2 Or any action if this set of positive actions is empty 5. Experiments and Results We evaluated the performance of the proposed model using the Pascal VOC dataset. The localization method is sensitive to the amount of data used for training, and for that reason, we used the combined training sets of 2007 and 2012. Using this joint set, results improved nearly 10% relative to using either one alone. We evaluate performance on the test set of VOC 2007 and report our findings under this setting. 5.1. Evaluation of Precision As an object detector, our algorithm can be evaluated in two modes: 1) All attended regions (AAR), a detector that scores all regions processed by the agent during a search episode. This is useful to consider well-localized regions that were not explicitly marked by the agent as detections. 2) Terminal regions (TR), a detector that only considers regions in which the agent explicitly used the trigger to indicate the presence of an object. In both cases, we use an external linear SVM trained with the same procedure as RCNN (with hard-negative mining on region proposals using the VOC2012 training set) to score the attended regions. This classifier is useful to rerank candidate regions assuming that our model generates object proposals. The scores computed by the Q-network are not useful for object detection evaluation because they estimate the value of actions instead of discriminative scores. We compare performance with other methods recently proposed in the literature. Table 1 presents detailed, percategory Average Precision (AP) for all systems. The only baseline method that does not use CNN features is the DPM system [11]. MultiBox [9] and DetNet [32] predict bounding boxes from input images using deep CNNs with a regression objective, and the work of Zou et al. [38] adapts the regionlets framework with CNN features too. Finally, we compare with the R-CNN system [12] configured with a network architecture that has the same number of layers as ours, and without bounding box regression. Overall, the R-CNN system still has the best performance and remains as a strong baseline. The major drawback of R-CNN is that it relies on a large number of object proposals to reach that performance, and demands significant computing power. MultiBox and Regionlets attempt to leverage deep CNNs in more efficient ways, by using only a few boxes to make predictions or avoiding re-computing features multiple times. Our approach also aims to localize objects by attending to a small number of regions, and this has an impact in performance. However, our result is significantly better than the other baseline methods, reaching 46.1 MAP, while the next best reaches 40.2 MAP. The main difference in performance between the TR and the ARR settings is that the first one only ranks regions explicitly marked by the agent with the trigger action. The Method DPM [11] MultiBox [9] DetNet [32] Regionlets [38] Ours TR Ours AAR R-CNN [12] aero 33.2 41.3 29.2 44.6 57.9 55.5 64.2 bike 60.3 27.7 35.2 55.6 56.7 61.9 69.7 bird 10.2 30.5 19.4 24.7 38.4 38.4 50.0 boat 16.1 17.6 16.7 23.5 33.0 36.5 41.9 bottle 27.3 3.2 3.7 6.3 17.5 21.4 32.0 bus 54.3 45.4 53.2 49.4 51.1 56.5 62.6 car 58.2 36.2 50.2 51.0 52.7 58.8 71.0 cat 23.0 53.5 27.2 57.5 53.0 55.9 60.7 chair 20.0 6.9 10.2 14.3 17.8 21.4 32.7 cow 24.1 25.6 34.8 35.9 39.1 40.4 58.5 table 26.7 27.3 30.2 45.9 47.1 46.3 46.5 dog 12.7 46.4 28.2 41.3 52.2 54.2 56.1 horse 58.1 31.2 46.6 61.9 58.0 56.9 60.6 mbike 48.2 29.7 41.7 54.7 57.0 55.9 66.8 person 43.2 37.5 26.2 44.1 45.2 45.7 54.2 plant 12.0 7.4 10.3 16.0 19.3 21.1 31.5 sheep 21.1 29.8 32.8 28.6 42.2 47.1 52.8 sofa 36.1 21.1 26.8 41.7 35.5 41.5 48.9 train 46.0 43.6 39.8 63.2 54.8 54.7 57.9 tv 43.5 22.5 47.0 44.2 49.0 51.4 64.7 MAP 33.7 29.2 30.5 40.2 43.9 46.1 54.2 Table 1. Average Precision (AP) per category in the Pascal VOC 2007 test set. The DPM system is the only baseline that does not use CNN features. R-CNN is the only method that uses object proposals. Our system is significantly better at localizing objects than other recent systems that predict bounding boxes from CNN features without object proposals. Numbers in bold are the second best result per column, and underlined numbers are the overall best result. 1 0.8 0.7 recall at IoU threshold 0.50 No action required for 11% of the detections 270 Bing EdgeBoxes Ours SelectiveSearch Gaussian Sliding window Uniform Frequency 0.9 612 200 Median: 11 5,147 total detections 94% of the distribution Mean: 25.6 100 0.6 0.5 0 0.4 25 50 Number of actions 75 100 Figure 5. Distribution of detections explicitly marked by the agent as a function of the number of actions required to reach the object. For each action, one region in the image needs to be processed. Most detections can be obtained with about 11 actions only. 0.3 0.2 0.1 0 0 10 0 1 10 2 10 # proposals 3 10 4 10 Figure 4. Recall as a function of the number of proposed regions. Solid lines are state-of-the-art methods, and dashed lines are simple baselines. Our approach is significantly better at early recall: only 10 proposals per image reach 50% recall. The tendency is also fundamentally different: overall recall does not depend strongly on a large number of proposed regions. Notice that our approach is not a category-independent region proposal algorithm. difference in performance is 2.2 percent points, which is relatively small, considering that TR proposes an average of only 1.3 regions per category per image (including true and false positives in all test images). Both settings require the same computational effort3 , because the agent has to attend the same number of regions. However, TR is able to identify which of those regions are promising objects with very high accuracy. 5.2. Evaluation of Recall All the regions attended by the agent can be understood as object proposal candidates, so we evaluate them following the methodology proposed by Hosang et al. [15]. Overall, our method running for 200 steps per category, pro3 The cost of the classifier is negligible compared to the cost of feature extraction and analysis. cesses a total of 4,000 candidates per image reaching 71% of recall. This is between 10 and 25 points less recall than most methods achieve at a similar number of proposals. However, the recall of our proposals is significantly superior for the top 100 candidates per image. For this evaluation we score attended regions using the Q-values predicted by the agent, and we add a large constant only to those regions for which the agent used the trigger. This gives priority to regions that the agent considers correctly localized objects. We use this scoring function instead of the classification scores for fairness in the evaluation [4], since other methods rank proposals using an estimated objectness score, and high Q-values can be interpreted in the same way. Figure 4 shows the number of proposals vs. recall for several baseline proposal methods. In this evaluation, we include selective search [34], BING [5] and EdgeBoxes [37]. The results show that our method reaches 50% of recall with just 10 proposals per image, while all the other methods are around or below 30%. This evaluation emphasizes two important points: first, our method uses category-specific knowledge to find objects, and this is clearly an advantage over categoryindependent proposals, but is also a drawback since objects that cannot be recognized given the feature representation are never localized. Second, the main challenge of our approach is to improve overall recall, that is, to localize ev- Intersection Over Union 1.0 1 3 4 2 0.5 2 3 0.0 35 45 82 4 141 1 Intersection Over Union Time (actions) 1.0 1 4 2 3 0.5 0.0 3 9 23 86 2 179 Time (actions) 5.3. Qualitative Evaluation We present a number of example sequences of regions that the agent attended to localize objects. Figure 7 shows two example scenes with multiple objects, and presents green boxes where a correct detection was explicitly marked by the agent. The plot to the left presents the evolution of IoU as the agent transforms the bounding box. These plots show that correct detections are usually obtained with a small number of steps increasing IoU with the ground truth rapidly. Points of the plots that oscillate below the minimum accepted threshold (0.5) indicate periods of the search process that were difficult or confusing to the agent. Figure 6 shows sequences of attended regions as seen by the agent, as well as the actions selected in each step. Notice that the actions chosen attempt to keep the object in the center of the box, and also that the final object appears to have normalized scale and aspect ratio. The top two exam- 3 Intersection Over Union ery single object without missing any instance. Notice that more candidates do not improve recall as much as other methods do, so we hypothesize that fixing overall recall will improve early recall even more. To illustrate this result further, we plot the distribution of correctly detected objects according to the number of steps necessary to localize them in Figure 5. The distribution has a long tail, with 83% of detections requiring less than 50 steps to be obtained, and an average of 25.6. A more robust statistic for long-tailed distributions is the median, which in our experiments is just 11 steps, indicating that most of the correct detections happen around that number of steps. Also, the agent is able to localize 11% of the objects immediately without processing more regions, because they are big instances that occupy most of the image. 1 Figure 7. Examples of multiple objects localized by the agent in a single scene. Numbers in yellow indicate the order in which each instance was localized. Notice that IoU between the attended region and ground truth increases quickly before the trigger is used. missed 1.0 1 3 2 0.5 error 2 0.0 7 32 200 Time (actions) correct 1 Intersection Over Union Figure 6. Example sequences observed by the agent and the actions selected to focus objects. Regions are warped in the same way as they are fed to the CNN. Actions keep the object in the center of the box. More examples in the supplementary material. Last row: example Inhibition of Return marks placed during test. 4 1.0 3 5 1 2 6 0.5 missed 4 0.0 15 33 100 123 6 147 5 2 1 error correct 4 3 200 Time (actions) Figure 8. Examples of images with common mistakes that include: duplicated detections due to objects not fully covered by the IoR mark, and missed objects due to size or other difficult patterns. ples also show the IoR mark that is placed in the environment after the agent triggers a detection. The reader can find more examples and videos in the supplementary material. 5.4. Error Modes We also evaluate performance using the diagnostic tool proposed by Hoiem et al. [14]. In summary, object localization is the most frequent error of our system, and it is sensitive to object size. Here we include the report of sensitivity to characteristics of objects in Figure 9, and compare to the R-CNN system. Our system is more sensitive to the size of objects than any other characteristic, which is mainly explained by the difficulty of the agent to attend cluttered re- R−CNN FT fc7: sensitivity and impact Agent AAR: Sensitivity and Impact 0.8 0.8 0.790 0.6 0.650 0.596 0.584 0.531 0.500 trn 0.685 0.442 0.429 0.335 0.292 0.208 occ 0.723 0.498 0.2 0.129 0 0.672 0.634 0.4 0.324 0.322 0.2 0.701 0.584 0.6 0.593 0.437 0.4 0.766 size asp view part 0 0.325 0.179 occ trn size asp view part Figure 9. Sensitivity analysis following the method of Hoiem et al. [14]. Error bars are the average normalized AP over categories with the highest and lowest performance for each characteristic. Evaluated characteristics are: occlusion (occ), truncation (trn), size, aspect ratio (asp), viewpoint of objects (view), and visibility of parts (parts). gions with small objects. Interestingly, our system seems to be less sensitive to occlusion and truncation than R-CNN. Figure 8 presents object search episodes with some common errors. The figure shows examples of objects that the agent tried to localize but IoU never passed the minimum threshold. Those objects are missed because none of the attended regions surpass the threshold. In both examples, the agent triggered an additional detection in a location where a correct detection was placed before. This happens because the inhibition-of-return (IoR) mark leaves part of the object visible that may be observed again by the agent. The IoR mark also generates other types of errors such as covering additional objects in the scene that cannot be recovered. We found that 78% of all missed detections happen in images with multiple instances of the same object, and the other 22% occur in images with a single instance. Not all missed objects are explained by obstruction of the IoR mark, however, we experimented with different strategies to change the focus of attention. We used a black box of the same size of the predicted box, but this results in a highly intrusive mark. We also experimented with a memory of detected instances to give negative rewards for going back to a known region, but training becomes unstable. The final cross that we adopted is centered in the predicted box and covers a third of its area, leaving parts of the region visible in favor of other overlapping objects. Overlapping objects are in general a challenge for most object detectors, and the problem requires special research attention. 5.5. Runtime We conducted runtime experiments using a workstation equipped with a K-40 GPU. Feature extraction and actiondecision run in the GPU. Our algorithm proceeds by analyzing a single region at a time, and processing each region takes an average of 7.74ms. This time is divided between feature extraction with the CNN (4.5 ms) and decision making with the Q-network (3.2ms), imposing an overhead of about 70% more computing power per region in our prototype, compared to systems that only extract features. This overhead is likely to be reduced using a more optimized implementation. Since we run the system for 200 steps, the total test processing time is 1.54s in average (wall clock time). Notice that we do not add segmentation or any other preprocessing time, because our system directly decides which boxes need to be evaluated. 6. Conclusions and Future Work A system that learns to localize objects in scenes using an attention-action strategy has been presented. This approach is fundamentally different from previous works for object detection, because it follows a top-down scene analysis to narrow down the correct location of objects. Reinforcement learning (RL) demonstrated to be an efficient strategy to learn a localization policy, which is a challenging task since an object can be localized following different search paths. Nevertheless, the proposed agent learns from its own mistakes and optimizes the policy to find objects. Experiments show that the system can localize a single instance of an object processing between 11 and 25 regions only. This results in a very efficient strategy for applications where a few number of categories are required. Our formulation needs to be extended in other ways for scaling to large numbers of categories; for instance, making it category-independent or using hierarchical ontologies to decide the fine-grained category later. An important challenge to improve recall needs to be addressed. Part of our future work includes training the system end-to-end instead of using a pre-trained CNN, and using deeper CNN architectures for improving the accuracy of predictions. Acknowledgements. Special thanks to David Forsyth and Derek Hoiem for discussions and advice, and to Oscar Sánchez for his feedback. We gratefully acknowledge NVIDIA corporation for the donation of Tesla GPUs for this research. This research was part of the Blue Waters sustained-petascale computing project. This work was partially supported by NSF grants IIS 1228082, CIF 1302438, and the DARPA Computer Science Study Group (D12AP00305). References [1] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004. 5 [2] J. Ba, V. Mnih, and K. Kavukcuoglu. Multiple object recognition with visual attention. arXiv:1412.7755, 2014. 2 [3] A. Borji and L. Itti. State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):185–207, 2013. 2 [4] N. Chavali, H. Agrawal, A. Mahendru, and D. Batra. Objectproposal evaluation protocol is’ gameable’. arXiv preprint arXiv:1505.05836, 2015. 6 [5] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. Torr. BING: Binarized normed gradients for objectness estimation at 300fps. In CVPR. IEEE, 2014. 6 [6] A. Coates, P. Abbeel, and A. Y. Ng. Learning for control from multiple demonstrations. In ICML, 2008. 5 [7] S. K. Divvala, D. Hoiem, J. H. Hays, A. Efros, M. Hebert, et al. An empirical study of context in object detection. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 1271–1278. IEEE, 2009. 2 [8] I. Endres, K. J. Shih, J. Jiaa, and D. Hoiem. Learning collections of part models for object recognition. In CVPR. IEEE, 2013. 2 [9] D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov. Scalable object detection using deep neural networks. In CVPR. IEEE, 2014. 1, 5, 6 [10] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2):303–338, 2010. 1 [11] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained partbased models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9):1627–1645, 2010. 2, 5, 6 [12] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR. IEEE, 2014. 1, 3, 5, 6 [13] A. Gonzalez-Garcia, A. Vezhnevets, and V. Ferrari. An active search strategy for efficient object class detection. In CVPR. IEEE, 2015. 2 [14] D. Hoiem, Y. Chodpathumwan, and Q. Dai. Diagnosing error in object detectors. In ECCV. Springer, 2012. 7, 8 [15] J. Hosang, R. Benenson, and B. Schiele. How good are detection proposals, really? arXiv:1406.6962, 2014. 2, 6 [16] L. Itti and C. Koch. Computational modelling of visual attention. Nature reviews neuroscience, 2(3):194–203, 2001. 3 [17] M. Juneja, A. Vedaldi, C. Jawahar, and A. Zisserman. Blocks that shout: Distinctive parts for scene classification. In CVPR. IEEE, 2013. 2 [18] C. H. Lampert, M. B. Blaschko, and T. Hofmann. Beyond sliding windows: Object localization by efficient subwindow search. In CVPR. IEEE, 2008. 2 [19] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. arXiv:1504.00702, 2015. 5 [20] J. J. Lim, H. Pirsiavash, and A. Torralba. Parsing ikea objects: Fine pose estimation. In ICCV. IEEE, 2013. 2 [21] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of exemplar-SVMs for object detection and beyond. In ICCV. IEEE, 2011. 2 [22] V. Mnih, N. Heess, A. Graves, et al. Recurrent models of visual attention. In NIPS, 2014. 2 [23] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518:529–533, 2015. 2, 4 [24] M. Pandey and S. Lazebnik. Scene recognition and weakly supervised object localization with deformable part-based models. In ICCV. IEEE, 2011. 2 [25] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497, 2015. 1 [26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014. 1 [27] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv:1312.6229, 2013. 1 [28] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. 1 [29] S. Singh, A. Gupta, and A. A. Efros. Unsupervised discovery of mid-level discriminative patches. In ECCV. Springer, 2012. 2 [30] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014. 5 [31] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 1998. 4 [32] C. Szegedy, A. Toshev, and D. Erhan. Deep neural networks for object detection. In NIPS, 2013. 5, 6 [33] A. Torralba, A. Oliva, M. S. Castelhano, and J. M. Henderson. Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review, 113(4):766, 2006. 2 [34] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. IJCV, 104(2):154–171, 2013. 6 [35] K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv:1502.03044, 2015. 2 [36] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV. Springer, 2014. 3 [37] C. L. Zitnick and P. Dollár. Edge boxes: Locating object proposals from edges. In ECCV. Springer, 2014. 6 [38] W. Y. Zou, X. Wang, M. Sun, and Y. Lin. Generic object detection with dense neural patterns and regionlets. arXiv:1404.4316, 2014. 5, 6 Deep Neural Decision Forests † Peter Kontschieder† , Madalina Fiterau , Antonio Criminisi† , Samuel Rota Bulò? ?  Fondazione Bruno Kessler Stanford University Microsoft Research Trento, Italy California, USA Cambridge, UK Abstract We present a novel approach to enrich classification trees with the representation learning ability of deep (neural) networks within an end-to-end trainable architecture. We combine these two worlds via a stochastic and differentiable decision tree model, which steers the formation of latent representations within the hidden layers of a deep network. The proposed model differs from conventional deep networks in that a decision forest provides the final predictions and it differs from conventional decision forests by introducing a principled, joint and global optimization of split and leaf node parameters. Our approach compares favourably to other state-of-the-art deep models on a large-scale image classification task like ImageNet. 1 Introduction Random forests [Amit and Geman, 1997; Breiman, 2001; Criminisi and Shotton, 2013] have found successfull application in machine learning in general and the computer vision community in particular. They empirically outperform most state-of-the-art learners on high dimensional data problems [Caruana et al., 2008], they are inherently able to deal with multi-class problems and are easily distributable on parallel hardware architectures. Moreover, they are considered to be close to an ideal learner [Hastie et al., 2009]. These facts and many (computationally) appealing properties make them attractive for various research areas and commercial products [Bosch et al., 2007; Brostow et al., 2008; Shotton et al., 2013]. On the downside, random forests lack a machanism to efficiently learn internal representations that help to capture the main factors of variation in the data [Bengio et al., 2010]. One of the consolidated findings of modern, deep learning approaches [Krizhevsky et al., 2012; Lin et al., 2013; Szegedy et al., 2014] is that their joint and unified way of learning internal data representations along with the classifiers greatly outperforms conventional feature descriptor & classifier pipelines on different tasks, given enough training data and computation capabilities (see e.g. [He et al., 2015] for image classification, [Yu and Deng, 2014] for speech recognition, [Fei-Fei, 2015] for image description). An interesting open question, which has received little attention in the literature so far, is how to endow random forests with the ability of learning proper internal representations of the input data in order to improve on the generalization capacity of the final classifier. Notable but limited exceptions are [Kontschieder et al., 2013; Montillo et al., 2013] where random forests were trained in an entangled setting, stacking intermediate classifier outputs with the original input data. The approach in [Rota Bulò and Kontschieder, 2014] introduced a way to integrate multi-layer perceptrons as split functions, however, representations were learned only locally at split node level and independently among split nodes. While these attempts can be considered early forms of representation learning in random forests, their prediction accuracies remained below the state-of-the-art. In this work we present Deep Neural Decision Forests – a novel approach to unify appealing properties from representation learning as known from deep architectures with the divide-and-conquer principle of decision trees. We introduce a stochastic, differentiable, and therefore back-propagation compatible version of decision trees, guiding the representation learning in hidden layers of deep networks. Thus, the task for representation learning is to reduce the uncertainty on the routing decisions of a sample taken at the split nodes, such that a globally-defined loss function is minimized. Additionally, for given split node parameters we obtain optimal predictions for all leaves of our trees by minimizing a convex objective, and we provide an optimization algorithm for it that does not depend on tedious step-size selection. Consequently, we can take the optimal decision for a test sample ending up in the leaves, with respect to all the training data and the current state of the network. We show the efficacy of our approach on the challenging ImageNet dataset for largescale image classification, where we obtain state-of-the-art results with no data augmentation. 2 Decision Trees with Stochastic Routing Decision trees can be used to tackle a wide range of learning problems [Criminisi and Shotton, 2013] including classification, which will be the focus of this work. Consider a classification problem with input and (finite) output spaces given by X and Y, respectively. A decision tree is a classifier consisting of decision (or split) nodes and prediction (or leaf) nodes organized into a tree structure. Decision nodes indexed d1 d2 d4 d3 d5 d6 d7 π `1 π `2 π `3 π `4 π `5 π `6 π `7 π `8 Figure 1: Routing of sample x along a decision tree to leaf `4 . Each node n ∈ N along the path steers the sample to the left with probability dn (x) and to the right with probability d¯n (x) = 1 − dn (x). So, the probability of reaching `4 is µ`4 (x) = d1 (x)d¯2 (x)d¯5 (x) and the related prediction is π `4 . by N are internal nodes of the tree, while prediction nodes indexed by L are the terminal nodes of the tree. A decision tree classifies a sample by routing it through the tree to a leaf node ` ∈ L, where the prediction takes place via a probability distribution π ` over Y. The routing along the tree is determined through decision functions dn (·; Θ) : X → [0, 1] parametrized by Θ and located in each decision node n ∈ N , which locally indicate the path to follow. When a sample x ∈ X reaches a decision node n it will be sent to the left or right subtree based on the output of dn (x; Θ). In standard decision forests, dn is binary and the routing is deterministic. In this paper we will consider instead a probabilistic routing, i.e. the routing direction is the output of a Bernoulli random variable with mean dn (x; Θ). Once a sample ends in a leaf node `, the related tree prediction is given by the class-label distribution π ` (see Fig. 1 for an illustration). In the case of stochastic routings, the leaf predictions will be weighted by the probability of reaching the leaf. Accordingly, the final prediction for sample x from tree T with decision nodes parametrized by Θ is given by X PT [y|x, Θ, π] = π`y µ` (x|Θ) , (1) `∈L where π = (π ` )`∈L and π`y denotes the probability of a sample reaching leaf ` to take on class y, while µ` (x|Θ) is regarded as the routing function providing P the probability that sample x will reach leaf `. Clearly, ` µ` (x|Θ) = 1 for all x ∈ X . To provide an explicit form for the routing function we introduce the following binary relations that depend on the tree’s structure: ` . n, which is true if ` belongs to the left subtree of node n, and n & `, which is true if ` belongs to the right subtree of node n. These relations allow us to express µ` as follows: Y (2) µ` (x|Θ) = dn (x; Θ)1`.n d¯n (x; Θ)1n&` , n∈N where d¯n (x; Θ) = 1−dn (x; Θ), and 1P is an indicator function conditioned on the argument P . Although the product in (2) runs over all nodes, only decision nodes along the path from the root to the leaf ` contribute to µ` (assuming 00 = 1), because for all other nodes 1`.n and 1n&` will be both 0. Decision nodes. We consider decision trees having decision functions of the following form to deliver stochastic routings: dn (x; Θ) = σ(fn (x; Θ)) , (3) where σ(x) = (1 + e−x )−1 is the sigmoid function, and fn (·; Θ) : X → R is a real-valued function depending on the sample and the parametrization Θ. The outcome of dn (x; Θ) ∈ [0, 1] represents the probability of sample x to be steered left in node n ∈ N . Further details about the functions fn can be found in Sec. 4, but intuitively depending on how we choose these functions we can model trees having shallow decisions (e.g. such as in oblique forests [Heath et al., 1993]) as well as deep ones. Forests of decision trees. A forest is an ensemble of decision trees F = {T1 , . . . , Tk }, which delivers a prediction for sample x by averaging the output of each tree, i.e. PF [y|x] = k 1X PTh [y|x] , k (4) h=1 omitting the tree parameters for notational convenience. 3 Learning Trees by Back-Propagation In order to learn a decision tree defined as per Sec. 2, we need to estimate both, the decision node parametrization Θ and the leaf predictions π. To carry out the estimation we follow the minimum empirical risk principle with respect to a given data set T ⊂ X × Y under log-loss, i.e. we pursue minimizers of the following risk term: 1 X R(Θ, π; T ) = L(Θ, π; x, y) , (5) |T | (x,y)∈T where L(Θ, π; x, y) is the log-loss term for the training sample (x, y) ∈ T , which is given by L(Θ, π; x, y) = − log(PT [y|x, Θ, π]) , (6) and PT is defined as in (1). We use a two-step optimization strategy, described in the rest of this section, alternating updates of Θ with updates of π in a way to minimize (5). Learning Decision Nodes. All decision functions depend on a common parameter Θ, which in turn parametrizes each function fn in (3). The minimization of the empirical risk with respect to Θ for a given π is, in general, a difficult and large-scale optimization problem, since no assumption has been made about the structure of fn . As an example, Θ could absorb all the parameters of a deep neural network having fn as one of its output units. For this reason, we will employ a Stochastic Gradient Descent (SGD) approach to minimize (5) with respect to Θ, as commonly done in the context of deep neural networks: ∂R (t) Θ(t+1) = Θ(t) − η (Θ , π; B) ∂Θ (7) η X ∂L (t) = Θ(t) − (Θ , π; x, y) . |B| ∂Θ (x,y)∈B Here, η > 0 is the learning rate and B ⊆ T is a random subset (a.k.a. mini-batch) of samples from the training set. Although not shown explicitly, we additionally consider a momentum term to smooth out the variations of the gradients. The gradient of the loss L with respect to Θ can be decomposed by the chain rule as follows X ∂L(Θ, π; x, y) ∂fn (x; Θ) ∂L (Θ, π; x, y) = . (8) ∂Θ ∂fn (x; Θ) ∂Θ n∈N Here, the gradient term that depends on the decision tree is ∂L(Θ, π; x, y) = dn (x; Θ)Anr − d¯n (x; Θ)Anl , ∂fn (x; Θ) (9) where nl and nr indicate the left and right child of node n, respectively, and we define Am for a generic node m ∈ N as P π`y µ` (x|Θ) , Am = `∈Lm PT [y|x, Θ, π] where Lm ⊆ L denotes the set of leaves held by the subtree rooted in node m. The actual computation of the gradient term in Eq. (9) can be carried out by traversing the tree twice [Kontschieder et al., 2015]. Learning prediction nodes. We consider now the minimization of (5) with respect to π when Θ is fixed, i.e. min R(Θ, π; T ) . (10) π This is a convex optimization problem and a global solution can be easily recovered. A similar problem has been encountered in the context of decision trees in [Rota Bulò and Kontschieder, 2014], but only at the level of a single node. In our case, however, the whole tree is taken into account, and we are jointly estimating all the leaf predictions. In order to compute a global minimizer of (10) we propose the following iterative scheme: (t+1) π`y = 1 (t) X 1y=y0 π`y µ` (x|Θ) (x,y 0 )∈T PT [y|x, Θ, π (t) ] (t) Z` , (11) (t) for all ` ∈ L and y ∈ Y, where Z` is a normalizing factor P (t+1) ensuring that y π`y = 1. The starting point π (0) can be arbitrary as long as every element is positive. A typical choice is to start from the uniform distribution in all leaves, (0) i.e. π`y = |Y|−1 . It is interesting to note that the update rule in (11) is step-size free and it guarantees a strict decrease of the risk at each update until a fixed-point is reached [Kontschieder et al., 2015]. Learning a forest. So far we have dealt with a single decision tree setting. Now, we consider an ensemble of trees F, where all trees can possibly share the same parameters in Θ, but each tree can have a different structure with a different set of decision functions (still defined as in (3)), and independent leaf predictions π. Since each tree in the forest F has its own set of leaf parameters π, we can update the prediction nodes of each tree independently via the update rule (11), given the current estimate of Θ. As for Θ, instead, we randomly select a tree in F for each mini-batch and then we proceed with the SGD update in (7), assuming that its leaf nodes’ parameters π are fixed. This strategy somewhat resembles the basic idea of Dropout [Srivastava et al., 2014], where each SGD update is potentially applied to a different network topology, which is sampled according to a specific distribution. In addition, updating individual trees instead of the entire forest reduces the computational load during training. At test time we average over tree predictions to produce the final outcome,(4). 4 Deep Decision Nodes We have defined decision functions dn in terms of real-valued functions fn (·; Θ), which are not necessarily independent, but coupled through the shared parametrization Θ. Our intention is to endow the trees with feature learning capabilities by embedding functions fn within a deep neural network with parameters Θ. To this end, we regard each function fn as a linear output unit of a deep network that will be turned into a probabilistic routing decision by the action of dn , which applies a sigmoid activation to obtain a response in the [0, 1] range. Fig. 2 provides a schematic illustration of this idea. The number of split nodes is determined by the number of output nodes of the preceding fully-connected layer. Under the proposed construction, the output units of the deep network are therefore not directly delivering the final predictions, e.g. through a Softmax layer, but each unit is responsible for driving the decision of a node in the forest. Indeed, during the forward pass through the deep network, a data sample x produces soft activations of the routing decisions of the tree that induce via the routing function a mixture of leaf predictions as per (1), which will form the final output. 5 Experiments on ImageNet ImageNet [Russakovsky et al., 2014] is a benchmark for large-scale image recognition tasks and its images are assigned to one out of 1000 possible ground truth labels. The dataset contains ≈1.2M training images, 50.000 validation images and 100.000 test images with average dimensionality of 482x415 pixels. Training and validation data are publicly available and we followed the commonly agreed protocol by reporting Top5-Errors on validation data. The GoogLeNet architecture [Szegedy et al., 2014], which has a reported Top5Error of 10.07% when used in a single-model, single-crop setting (see first row in Tab. 3 in [Szegedy et al., 2014]), served as basis for our experiments. As opposed to conventional architectures, GoogLeNet uses 3 Softmax layers at different stages of the network to encourage the construction of informative features, due to its very deep architecture. In order to obtain a Deep Neural Decision Forest architecture coined dNDF.NET, we have replaced each Softmax layer from GoogLeNet with a forest consisting of 10 trees (each fixed to depth 15), resulting in a total number of 30 trees. We refer to the individual forests as dNDF0 (closest to raw input), dNDF1 (replacing middle loss layer in GoogLeNet) and dNDF2 (as terminal layer). Further details about our dNDF.NET architecture are provided in [Kontschieder et al., Deep CNN with parameters Θ FC f4 f2 f5 f1 f6 f3 f7 f11 f9 f12 d1 π1 d3 d5 π2 π3 d9 d6 π4 f13 f10 f14 d8 d2 d4 f8 π5 d7 π6 π7 d11 π8 π9 d10 d12 π 10 π 11 d13 π 12 π 13 π 14 d14 π 15 π 16 Figure 2: Illustration how to implement a deep neural decision forest (dNDF). Top: Deep neural network with variable number of layers, represented via parameters Θ. FC block: Fully Connected layer used to provide functions fn (·; Θ) (here: inner products), described in Eq. (3). Each output of fn is brought in correspondence with a split node in a tree, eventually producing the routing (split) decisions dn (x) = σ(fn (x)). The order of the assignments of output units to decision nodes can be arbitrary (the one we show allows a simple visualization). The circles at the bottom correspond to leaf nodes, holding probability distributions π ` as a result of solving the convex optimization problem defined in Eq. (10). GoogLeNet [Szegedy et al., 2014] #models/#crops Top5-Errors 1/1 1/10 1/144 7/1 7/10 dNDF.NET 7/144 1/1 1/10 dNDF0 dNDF1 dNDF2 7/1 10.07% 9.15% 7.89% 8.09% 7.62% 6.67% 7.84% 7.08% 6.38% 1/1 9.26% 8.49% 7.92% Table 1: Top5-Errors obtained on ImageNet validation data, comparing our dNDF.NET to GoogLeNet. 2015]. Following the implementation guideline for decision nodes in Section 4, we randomly selected 500 output dimensions of the respectively preceding layers in GoogLeNet for each decision function fn . We trained the network for 1000 epochs using (mini-) batches composed of 100.000 images (which was feasible due to distribution of the computational load to a cluster of 52 CPUs and 12 hosts, where each host is equipped with a NVIDIA Tesla K40 GPU). of Tab. 1, we report also the performance of each individual forest dNDFx (x = 0, 1, 2) within the architecture. As expected, dNDF0 (closest to the input layer) performs worse than dNDF2 , which is the final layer of dNDF.NET, but only by 1.34%. Averaging over all three dNDF forest outputs yields the lowest Top5-Error of 7.84%. As for the posterior learning, we only update the leaf node predictions of the tree that also receives split node parameter updates, i.e. the randomly selected one as described in Section 3. To improve computational efficiency, we consider only the samples of the current mini-batch for posterior learning, while all the training data could be used in principle. However, since we use mini-batches composed of 100.000 samples, we can approximate the training set sufficiently well and, at the same time, introduce a positive, regularizing effect. In this paper we have shown how to model and train stochastic, differentiable decision trees, usable as alternative classifiers for end-to-end learning in (deep) convolutional networks. Prevailing approaches for decision tree training typically operate in a greedy and local manner, making representation learning impossible. To overcome this problem, we introduced stochastic routing for decision trees, enabling split node parameter learning via back-propagation. Moreover, we showed how to populate leaf nodes with their optimal predictors, given the current state of the tree/underlying network. We have successfully validated our new decision forest model on ImageNet and surpassed state-of-the-art when integrating it in the GoogLeNet architecture, without any form of dataset augmentation, further detailed in [Kontschieder et al., 2015]. Tab. 1 provides a summary of Top5-Errors on validation data for our proposed dNDF.NET against GoogLeNet. We ascribe the improvements on the single crop, single model setting (Top5-Error of only 7.84%) to our proposed approach, as the only architectural difference to GoogLeNet is deploying our dNDFs. By using an ensemble of 7 dNDF.NETs (still single crop inputs), we can reduce the error further and obtain a Top5-Error of 6.38%, which is better than the best result of 6.67%, obtained with 7 GoogLeNets using 144 crops per image [Szegedy et al., 2014]. In the last three columns 6 Conclusions Acknowledgments Peter and Samuel were partially supported by Novartis Pharmaceuticals and the ASSESS MS project. Madalina was partially supported by NSH Award 1320347, DARPA Contract FA8750-12-2-0324 and the Mobilize Center (NIH U54 EB020405). References [Amit and Geman, 1997] Y. Amit and D. Geman. Shape quantization and recognition with randomized trees. (NC), 9(7):1545–1588, 1997. [Bengio et al., 2010] Yoshua Bengio, Olivier Delalleau, and Clarence Simard. Decision trees do not generalize to new variations. Computational Intelligence, 26(4):449–467, 2010. [Bosch et al., 2007] A. Bosch, A. Zisserman, and X. Muñoz. Image classification using random forests and ferns. In (ICCV), 2007. [Breiman, 2001] Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001. [Brostow et al., 2008] Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In (ECCV). Springer, 2008. [Caruana et al., 2008] R. Caruana, N. Karampatziakis, and A. Yessenalina. An empirical evaluation of supervised learning in high dimensions. In (ICML), pages 96–103, 2008. [Criminisi and Shotton, 2013] A. Criminisi and J. Shotton. Decision Forests in Computer Vision and Medical Image Analysis. Springer, 2013. [Fei-Fei, 2015] Andrej Karpathy Li Fei-Fei. Deep visualsemantic alignments for generating image descriptions. In (CVPR), 2015. [Hastie et al., 2009] T. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Statistical Learning. Springer, 2009. [He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR, abs/1502.01852, 2015. [Heath et al., 1993] David Heath, Simon Kasif, and Steven Salzberg. Induction of oblique decision trees. Journal of Artificial Intelligence Research, 2(2):1–32, 1993. [Kontschieder et al., 2013] P. Kontschieder, P. Kohli, J. Shotton, and A. Criminisi. GeoF: Geodesic forests for learning coupled predictors. In (CVPR), pages 65–72, 2013. [Kontschieder et al., 2015] P. Kontschieder, M. Fiterau, A. Criminisi, and S. Rota Bulò. Deep neural decision forests. In (ICCV), 2015. [Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In (NIPS), 2012. [Lin et al., 2013] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. CoRR, abs/1312.4400, 2013. [Montillo et al., 2013] A. Montillo, J. Tu, J. Shotton, J. Winn, J. E. Iglesias, D. N. Metaxas, and A. Criminisi. Entangled forests and differentiable information gain maximization. In Decision Forests in Computer Vision and Medical Image Analysis. Springer, 2013. [Rota Bulò and Kontschieder, 2014] Samuel Rota Bulò and Peter Kontschieder. Neural decision forests for semantic image labelling. In (CVPR), 2014. [Russakovsky et al., 2014] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2014. [Shotton et al., 2013] Jamie Shotton, Ross Girshick, Andrew Fitzgibbon, Toby Sharp, Mat Cook, Mark Finocchio, Richard Moore, Pushmeet Kohli, Antonio Criminisi, Alex Kipman, and Andrew Blake. Efficient human pose estimation from single depth images. (PAMI), 2013. [Srivastava et al., 2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014. [Szegedy et al., 2014] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. [Yu and Deng, 2014] D. Yu and L. Deng. Automatic Speech Recognition: A Deep Learning Approach. Springer, 2014. Im2Calories: towards an automated mobile vision food diary Austin Myers, Nick Johnston, Vivek Rathod, Anoop Korattikara, Alex Gorban Nathan Silberman, Sergio Guadarrama, George Papandreou, Jonathan Huang, Kevin Murphy amyers@umd.edu, (nickj, rathodv, kbanoop, gorban)@google.com (nsilberman, sguada, gpapan, jonathanhuang, kpmurphy)@google.com We present a system which can recognize the contents of your meal from a single image, and then predict its nutritional contents, such as calories. The simplest version assumes that the user is eating at a restaurant for which we know the menu. In this case, we can collect images offline to train a multi-label classifier. At run time, we apply the classifier (running on your phone) to predict which foods are present in your meal, and we lookup the corresponding nutritional facts. We apply this method to a new dataset of images from 23 different restaurants, using a CNN-based classifier, significantly outperforming previous work. The more challenging setting works outside of restaurants. In this case, we need to estimate the size of the foods, as well as their labels. This requires solving segmentation and depth / volume estimation from a single image. We present CNN-based approaches to these problems, with promising preliminary results. 1. Introduction Many people are interested in tracking what they eat to help them achieve weight loss goals or manage their diabetes or food allergies. However, most current mobile apps (MyFitnessPal, LoseIt, etc) require manual data entry, which is tedious and time consuming. Consequently, most users do not use such apps for very long [9]. Furthermore, amateur self-reports of calorie intake typically have an error rate that exceeds 400 calories per day [31, 5]. Rather than rely purely on data entry, several approaches make use of a mobile camera to aid in this task. Cordeiro et al. [8] records a photo of the meal but does not perform any visual analysis of the image. Several previous approaches [23] [29] rely on an expert nutritionists to analyse the image offline (at the end of each day). Other approaches [26] [25] use crowd sourcing to interpret the image, in lieu of an expert. However, crowd sourcing is both costly and slow, which hinders widespread adoption. Several existing works [39, 21, 37] do use computer vision algorithms to reason about meals but only work in laboratory conditions where the food items are well separated and the number of categories is small. Furthermore, most of these methods use traditional, hand-crafted visual features, and only use machine learning at the classification stage. The holy grail is an automatic method for estimating the nutritional contents of a meal from one or more images. Unfortunately, even a perfect visual interpretation of the scene cannot perfectly predict what is inside many foods, e.g., a burrito. Consequently, an ideal system is one which correctly infers what is knowable and prompts the user for feedback on inherantly ambiguous components of a meal. Our goal is to minimize user effort for completing food diaries by offering smart ”auto-complete” functionality, rather than complete automation. In this paper, we take some initial steps towards such a system. Our approach utilizes several deep learning algorithms, tailored to run on a conventional mobile phone, trained to recognize food items and predict the nutrional contents meals from images taken “in the wild”. We refer to this task as the “Im2Calories” problem, by analogy to the recent line of work on the “Im2Text” problem. It should be stressed, however, that we are interested in estimating various other properties of a meal (such as fat and carbohydrates) and not just calories. We start by building on [1], who developed a system that can predict the calorie content of a meal from a single image. Their key insight (also made in [2]) was that the problem becomes much simpler if we restrict the setting to one where the user is eating in a restaurant whose menu is known. In this case, the problem reduces to one of detecting which items, out of the K possible items on the menu, the user has chosen. Each item typically has a standard serving size1 , and we typically know its nutritional contents.2 , whereas getting nutritional information for arbitrary cooked foods is much harder, as discussed in Section 7. In Section 3, we show that by simply replacing the handcrafted features used in [1] with a convolutional neural network (CNN), we can significantly improve performance, 1 There may be variants, but these are typically few in number (e.g., small, medium or large fries). Hence we an treat these as different items. 2 The US Food and Drug Administration has passed a law requiring all major chain restaurants to post the nutritional contents of their meals, starting in December 2016. See [15] for details. both at labeling the foods and estimating total calories. We then extend their method from 3 restaurants to 23 restaurants, increasing the coverage from 41 food items to 2517. We show that the same basic multilabel classification system continues to work. Unfortunately, it is hard to get enough images for all the menu items for all the restaurants in the world. And even if we could, this would not help us when the user is not eating at a restaurant. Therefore, in Section 4, we develop a set of 201 generic, restaurant-independent food tags. We then extend the existing public Food101 dataset [3] with these tags using crowdsourcing. We call the resulting dataset Food201-multilabel and plan to release it publicly.3 We show that the same kind of CNN-based multi-label classifier also works fairly well on this new (larger and more challenging) dataset, although we found it necessary to perform some clustering of visually indistinguishable labels in order to get acceptable performance. Of course, detecting the presence of a food item in an image is not sufficient, since most items can be “parameterized” in terms of size, toppings, etc. Note that this is true even in the restaurant setting. We could of course ask the user about these variants, but we would like to do as much as possible automatically. To be able to perform such fine grained classification, we need to be able to localize the objects within the image and extract detailed features. We argue that a segmentationbased approach is more appropriate than the traditional bounding-box approach, since food items can have highly variable shape. In Section 5, we present the new Food201segmented dataset, and our approach to semantic image segmentation. We show that leveraging the multilabel classifier from the earlier stage can help with segmentation, since it provides a form of “context”. Once we have segmented the foods, we can try to estimate their volume. To do this, we need to know the surface height of the foods above the plate. In Section 6, we present some preliminary results on estimating the depth of each pixel from a single RGB image, using a CNN. We then show promising results on estimating the volumes of foods. In summary, this paper makes 3 main contributions. First, we develop a system that can recognize the contents of a restaurant meal much more accurately than the previous state of the art, and at a much larger scale. Second, we introduce a new dataset, Food201, and show how it can be used to train and test image tagging and segmentation systems. Third, we show some promising preliminary results on the challenging problem of mapping image to calories from images taken in the wild, in a non-restaurant setting. Our overall system is illustrated in Figure 1. 3 See https://storage.googleapis.com/food201/ food201.zip. Figure 1. Illustration of the overall system. Dotted boxes denote components that have not yet been implemented. The input is one or more images, and optionally a GPS signal and user priors (e.g., concerning food preferences). The output is a food diary, and an estimated total calorie count (in cases where a suitable nutritional database is available, such as from a restaurant’s menu). Name Food101 Food101-Background Food201-MultiLabel Food201-Segmented Restaurant Gfood-3d Nfood-3d #Classes 101 2 201 201 2517 - #Train 75,750 151,500 35,242 10,100 75k 150k - #Test 25,250 50,500 15,132 2,525 25k 2471 1050 Comments [3] Food vs non-food Multi label Label per pixel Single label Depth per pixel Depth per pixel Table 1. Summary of the datasets used in this paper. The Restaurant dataset contains web images for 23 restaurants. Gfood-3d is from 50 Google meals. Nfood-3d is from 11 meals made with Nasco food replicas. 2. Meal detection The first step in our pipeline is to determine if the image is an image of a meal at a “reasonable” distance from the camera. We can phrase this as a simple binary classification problem. To tackle this problem, we need to have a suitable dataset. We start by taking the Food101 dataset developed in [3], which contains 101 classes of food, 1000 images each (750 train, 250 test). Food101 is the largest publicly available dataset of food images we are aware of (see Table 1 for a comparison with some other public datasets). The Food101 dataset is designed for multi-class classification. To make a dataset suitable for binary classification, we combined all the food classes into one generic “food” class, and then randomly extracted an equal number of images from the ImageNet challenge dataset [30] to create the “non-food” class.4 Each image is rescaled (if necessary) so that its maximum height or width is 512 pixels, but preserving the original aspect ratio. We call this new dataset the Food1014 ImageNet actually contains 51 food classes, so we removed these images from the negative set. Background dataset. To train a classifier for this problem, we took the GoogLeNet CNN model from [34], which had been pretrained on ImageNet, removed the final 1000-way softmax, and replaced it with a single logistic node. Then we fine tune the whole model on the Food101-Background dataset; this takes about 12 hours using a single Titan X GPU with 12 GB of memory. The final test set accuracy is 99.02%. Method Baseline Meal Snap Menu Match Ĉ C̄ Mean error −37.3 ± 3.9 −268.5 ± 13.3 −21.0 ± 11.6 −31.90 ± 28.10 −25.35 ± 26.37 Mean absolute error 239.9 ± 1.4 330.9 ± 11.0 232.0 ± 7.2 163.43 ± 16.32 152.95 ± 15.61 Table 2. Errors in calorie estimation on the MenuMatch dataset. Ĉ and C̄ are our methods. Numbers after the ± sign are standard errors estimated by 5-fold cross-validation. See text for details. 3. Restaurant-specific im2calories Once we have determined that the image contains a meal, we try to analyze its contents. The first step is to determine which restaurant the user is in. In this paper, we use Google’s Places API [27] for this. We then retrieve the menu of the nearest restaurant from the web,5 parse the menu into a list of K food items, and retrieve images for each of these, either by performing web search (as in [2]) or by asking users to collect images (as in [1]).6 Once we have the dataset, we can train a classifier to map from image to label. Since the user may have multiple food items on their plate, it is better to use a multi-label classifier rather than using a multi-class classifier, which assumes the labels are mutually exclusive. Next we can estimate the set of foods that are present by picking a suitable threshold φ, and then computing S = {k : p(yk = 1|x) > φ}, where p(yk = 1|x) is the probability that food k is present in image x. Finally, we can lookup each of these food items in our (restaurant specific) database, P and then estimate the total calories as follows: Ĉ = k∈S Ck , where Ck is the calorie content of menu item k. Alternatively, to avoid P having to specify the threshold φ, we can compute K C̄ = k=1 p(yk = 1|x)Ck . We compare these methods below. 3.1. Experiments on MenuMatch dataset To evaluate this approach, we used the dataset from [1], known as “MenuMatch”. This consists of 646 images, tagged with 41 food items, taken from 3 restaurants. Unlike other datasets, each image in MenuMatch has a corresponding ground truth calorie estimate, computed by an expert nutritionist. In addition, each restaurant’s menu has corresponding ground truth calorie content per item. [1] used various hand-crafted features together with a linear SVM, trained in a one-vs-all fashion. Their best performing system achieved a mean average precision (mAP) 5 This data is available for many US chain restaurants in semistructured form from https://www.nutritionix.com. 6 In practice, it is suprisingly complicated to parse the menus and retrieve relevant images. For example, the restaurant “16 handles” contains the folowing items: NSA Blueberry Tease, NSA Chocolate Eruption, NSA Strawberry Fields, and 65 other similar entries. You need to know that these are from the frozen yogurt section of the menu, and that NSA stands for “no sugar added”, in order to make any sense of this data. of 51.2% on the test set. By contrast, we get much better results using a deep learning approach, as we explain below. We took the GoogLeNet CNN model from [34], which had been pre-trained on ImageNet, removed the final 1000way softmax, replaced it with a 101-way softmax, and finetuned the model on the Food101 dataset [3]. The resulting model has a classification accuracy on the Food101 test set of 79%, which is significantly better than the 50.76% reported by [3] (they used hand crafted features plus an SVM classifer using a spatial pyramid matching kernel). Next, we took the model which was trained on Food101, removed the 101-way softmax, replaced it with 41 logistic nodes, and fine-tuned on the MenuMatch training set. (Pretraining on Food101 was necessary since the MenuMatch dataset is so small.) The resulting mAP on the test set is 81.4%, which is significantly higher than the best result of 51.2% reported in [1]. Finally, we wanted to assess the accuracy of calorie prediction. We compared 5 methods: the MenuMatch system of [1], the MealSnap app [25] that uses crowdsourcing, our method using Ĉ, our method using C̄, and finally, a baseline method, which simply computes the empirical mean of the calorie content of all meals from a specific restaurant. The results are shown in Table 2. We see that our system has considerably lower error than MenuMatch and the crowdsourced MealSnap app. (The unreliability of MealSnap was also noted in [26].) In fact, we see that MenuMatch barely beats the baseline approach of predicting the prior mean. 3.2. Scaling up to more restaurants The MenuMatch results are based on 646 images of 41 food items from 3 restaurants. In this Section, we discuss our attempts to scale up these experiments. First, we downloaded the menus for the top 25 restaurants in the USA, as ranked by sales.7 We decided to drop “Pizza Hut” and “Chipotle”, since gathering images for their menu items was tricky.8 From the remaining 23 7 Source: http://nrn.com/us-top-100/ top-100-chains-us-sales. 8 For example, “Pizza Hut” has menu items such as “chicken”, “pepperoni”, etc. But these are toppings for a pizza, not individual food items. Similarly, “Chipotle” lists “chicken”, “beans”, etc. But these are fillings for a burrito, not individual food items. Figure 3. The first two images are put into the same visual cluster, the third is kept distinct. Image sources: http://www.mcdonaldsmenu.mobi/beefburgersmenu/deluxequarterpounder/. http://www.mcdonaldsmenu.mobi/beefburgersmenu/baconcheesequarterpounder/. http://www.mcdonalds.com/us/en/food/product_nutrition.burgerssandwiches.7. quarter- pounder- with- cheese.html Figure 2. Top 1 and top 5 error rates on the test set for 23 different restaurants, before and after clustering the most confusable labels. restaurants, we collected 4857 menu items. We manually removed drinks and other miscellaneous items, and then scraped 1.2 million images by issuing image search queries of the form “<restaurant name> <item name> (yelp | flickr | instagram | pinterest | foodspotting)”. (These site restricts were chosen to increase the chances that we collected user-generated photos, rather than “official” photos from the restaurant itself, which tend to be too easy.) Of the scraped images, 270k were sent to Amazon Mechanical Turk for verification, producing a final set of 2517 menu items and 99k images. We call this the Restaurant dataset. We then took our GoogleLeNet model, which had been trained on ImagetNet and then Food101, and replaced the 101-softmax with 2517 logistic nodes. We trained the final layer of this model on 75% of the data, and tested on the rest. We then computed the top 1 and top 5 error rates, averaged over the food items, for each of the 23 restaurants. The results are shown in Figure 2. The top 1 error rate is quite high. This is because many food items are extremely similar, and it is hard, even for a human expert, to tell them apart. For example, McDonalds has the following items on its menu: Quarter Pounder Deluxe, Quarter Pounder Bacon Cheese, Quarter Pounder with Cheese, etc. (See Figure 3 for an illustration of these food items.) To combat this issue, we computed the class confusion matrix on the training set for each restaurant, and then performed a very simple clustering of similar items. In particular, we computed the K nearest neighbor graph, in which we connected each label to the K other labels it is most often mapped to (which can include itself). We then computed the connected components to form a set of clusters. We found qualitatively that using K = 1 gave a good tradeoff between merging the most confusable classes and overclustering. With K = 1, the number of clusters was about 0.9 times the original number of items; most clusters were singletons, but some had 2 or 3 items in them. Figure 3 gives an example of two clusters we created from the McDonald’s menu; the first cluster contains two very visually similar items (Quarter Pounder Deluxe and Quarter Pounder Bacon Cheese), and the second cluster contains a visually distinctive item (Quarter Pounder with Cheese). Finally, we evaluated performance of the classifier on the clustered test labels, by defining the probability of a cluster as the max probability of any label in the cluster. Figure 2 shows that, not surprisingly, the error rates decrease. In the future, we hope to try using a hierarchical classifier, which can tradeoff specificity with error rate c.f., [11]. 4. Generic food detection The results from Section 3 required images for each menu item from each restaurant. It is difficult to acquire such data, as previously remarked. To create a more generic dataset, we took half of the Food101 dataset (50k images), and asked raters on Mechanical Turk to name all the food items they could see in each image. We included the original class label as one of the choices, and manually created a list of commonly co-occuring foods as additional choices in the drop-down menu (e.g., eggs often co-occur with bacon). We also allowed raters to enter new terms in a text box. We used 2 raters per image. After manually merging synonymous terms, and removing terms that occurred less than 100 times, we ended up with a vocabulary of 201 labels. On average, each image had 1.9 labels (with a median of 1 and a max of 8). We split the resulting dataset into 35,242 training images and 15,132 test images, in a way which is consistent with the Food101 train/ test split. We call this the Food201MultiLabel dataset. See Table 1 for a summary of how this dataset compares to other public food datasets. Next, we developed a multi-label classifier for this Figure 4. Precision-recall curves for 6 classes, ranging from best to worst, on the Food201-MultLabel dataset. dataset, using the same method as in Section 3 (namely taking the GoogLeNet model, replacing the 101-way softmax with 201 logistic nodes). This takes about half a day to train on a single GPU. We then compute the average precision (area under the precision-recall curve) for each class, and average this over the classes, to compute the mean average precision (mAP). The mAP is 0.8 for classes in Food 101, 0.2 for classes outside Food 101, and 0.5 overall. Not surprisingly, we do better on the original Food101 classes, since the new classes often correspond to side dishes or smaller food items, and are much less frequent in the training data. The top 3 classes were: edamame (0.987), macarons (0.976), hot and sour soup (0.956). The bottom 3 classes were: cream (0.015), garnish (0.014), peppers (0.010). Figure 4 shows the precision-recall curves for some of these classes. 5. Semantic image segmentation In addition to predicting the presence of certain foods, it is useful to localize them in the image. Since most foods are amorphous, it is better to segment out the region corresponding to each food, rather than putting a bounding box around them. Such a segmented image will enable further analysis, such as counting and size estimation (see below), which is essential for nutrition estimation. We can also allow the user to interactively edit the estimated segmentation mask, in order to improve accuracy of the system (although we leave this to future work). To train and evaluate semantic image segmentation systems, we took a subset of the Food201-MultiLabel dataset (12k images), and asked raters on a crowd computing platform to manually segment each of the food items that have been tagged (in an earlier stage) in that each image. We used 1 rater per image, and an internal tool that leverages grab- cut to make this labeling process reasonably fast. Raters had the option of skipping foods that were too hard to segment. Thus some foods are not segmented, resulting in false negatives. We call this the Food201-segmented dataset. Note that we are segmenting each class, as in the PASCAL VOC semantic segmentation challenge [13]. Arguably, instance-level segmentation (see e.g., [17]) would be more useful, since it distinguishes different instances of the same class, which would enable us to count instances.9 . However, this is quite difficult. For example, consider distinguishing pancake 1 from pancake 2 in the food image in Figure 1: this is quite challenging, since the top pancake almost completely covers the bottom one. Furthemore, segmenting the eggs into 2 instances is even harder, since the boundary is not well defined. We leave instance-level segmentation to future work. To tackle the segmentation problem, we use the “DeepLab” system from [6].10 This model uses a CNN to provide the unary potentials of a CRF, and a fully connected graph to perform edge-sensitive label smoothing (as in bilateral filtering). The model is initialized on ImageNet, and then finetuned on Food201-segmented, which takes about 1 day on a single GPU. For the 3 CRF parameters, which control the strength of the edge potentials, we use the parameter values from [6]; these were chosen by grid search, to minimize validation error on held-out training data. The label set in the Food201-Segmented dataset is much larger than in the VOC challenge (201 labels instead of just 20). Consequently, the baseline model has a tendency to generate a lot of false positives. To improve performance, we take the probability vector p(yk = 1|x) computed by the multi-label classifier from Section 4, find the top 5 entries, and then create a binary mask vector, which is all 0s except for the top 5 labels, plus the background. We then multiply the per-pixel label distribution from the segmentation CNN by this sparse vector, before running the CRF smoothing. This provides a form of global image context and improves the results considerably (see below). We show some sample results in Figure 5. Consider the last row, for example. We see that the context from the multilabel model helps by eliminating false positives (e.g., caprese salad and caesar salad). We also see that the ground truth is sometimes incomplete (e.g., the burrito is not actu9 It is more natural for the user if the system records in their food diary that they ate 3 slices of pizza rather than, say, 120 ounces of pizza. It is also much easier for the user to interactively fix errors related to discrete counts than continuous volumes. Of course, this assumes we know what the size of each slice is. We leave further investigation to future work. 10 At the time this paper was submitted, DeepLab was the best performing method on the PASCAL VOC challenge (see http://host.robots.ox.ac.uk:8080/leaderboard/ displaylb.php?challengeid=11&compid=6). At the time of writing the camera ready version, the best performing method is an extension of DeepLab that was additionally trained on MS-COCO data. 6. Volume estimation Figure 5. Examples of semantic image segmentation on some images from the Food201-Segmented test set. First column: original image. Second column: ground truth. Third column: predictions using CNN/CRF. Fouth column: predictions using CNN/CRF with multilabel context. Best viewed in color. CRF? Context? Acc Recall IoU 0 0 0.71 0.30 0.19 1 0 0.74 0.32 0.22 0 1 0.74 0.32 0.23 1 1 0.76 0.33 0.25 Table 3. Performance on the Food101-Segmented test set. ally labeled by the human). Finally, we see that the label space is somewhat arbitrary: the ground truth uses the term “salad”, whereas the model predicts “lettuce”. Currently we ignore any semantic similarities between the labels. The performance of our system with and without the CRF, and with and without multilabel context, is shown in Table 3. The performance in terms of IoU is much lower than on the PASCAL VOC 2012 segmentation challenge. We believe this is due to several factors: (1) we have 201 foreground labels to predict, whereas VOC just has 20; (2) our labeled dataset suffers from incompleteness, which can result in correct predictions being erroneously being counted as false positives, as well as “polluting” the background class during training; (3) our task is arguably intrinsically harder, since the categories we wish to segment are more deformable and varied in their appearance than most of the VOC categories. Having segmented the foods, the next step is to estimate their physical size (3d volume). We first predict the distance of every pixel from the camera, using the same CNN architecture as in [12] applied to a single RGB image. We trained our model on the NYU v2 RGBD dataset of indoor scenes, and then fine tuned it on a new 3d food dataset we collected which we call the GFood3d dataset (G for Google). This consists of short RGBD videos of 50 different meals from various Google cafes collected using the Intel RealSense F200 depth sensor.11 In total, the dataset has about 150k frames. (Note that each RGB image has a corresponding depth image, but otherwise this is unlabeled data.) On a test set of 2,471 images (recorded in a different set of Google cafes), we get an average relative error of 0.18 meters, which is slightly better to the performance reported in [12] on the NYU dataset. Figure 6 shows a qualitative example. We see that the CNN is able to predict the depth map fairly well, albeit at a low spatial resolution of 75 x 55. (For comparison with the F200 data, we upsample the predicted depth map). The next step is to convert the depthmap into a voxel representation. To do this, we first detect the table surface using RANSAC. Next, we project each pixel into 3d space, exploiting the known intrinsic parameters of the F200 sensor. Finally, we tile the table surface with a 2d grid (using a cell size of 2mm x 2mm), and compute the average height of all the points in each cell, thus creating a “tower” of that height. For an example of the result of this process, see Figure 7. Given a voxel representation of the food, and a segmentation mask, we can estimate the volume of each food item. To measure the accuracy of this approach, we purchased the “MyPlate” kit from Nasco.12 This contains rubber replicas of 42 food items in standard sizes, and is used for training nutritionists. We verified the actual size of these items using the water displacement method (we found that the measured size was somewhat smaller than the quoted sizes). We then created 11 “meals” from these food replicas, and recorded images of them using the F200. Specifically, we put the meals on a turntable, and automatically took 100 images as the plate went through a full rotation. The resulting dataset has 1050 depth images. We call this the NFood-3d dataset (N for Nasco). To measure performance of our system, we compute the absolute error in the volume estimate. We can estimate the volume using the true depth map (from F200) or the pre11 The F200 has a spatial resolution of 1920 x 1080 pixels, and a sensing range of 0.2–1.2m. The Kinect2 camera has the same resolution, but a sensing range of 0.8–3.5m. Thus the F200 is a better choice for close up images. 12 Source: http://www.enasco.com/product/WA29169HR. Figure 6. (a) An image from the GFood-3d test set. (b) Depth recorded from RealSense RGBD sensor. (c) Depth predicted by CNN. Figure 7. (a) An image from the NFood-3d dataset. (b) Voxel grid derived from RealSense depthmap. (c) Voxel grid estimated from CNN predicted depthmap. Size of grid cell is 2mm x 2mm. 7. Calorie estimation Figure 8. Absolute error in volume estimation (in ml) across the 11 meals in the NFood-3d dataset. Each meal has 100 images, taken from different viewing angles. The boxplots plot the distribution of errors across these 100 images. dicted depth map (from CNN), and using the true segmentation mask (from human) or the predicted segmentation mask (from CNN). Unfortunately, we have found that our segmentation system does not work well on the NFood images, because their color and texture properties are too dissimilar to real food. However, we were able to compare the quality of using the true depth map and predicted depth map. The results are shown in Figure 8. We see that for most of the meals, our CNN volume predictor is quite accurate. The final step is to map from the volume to the calorie content. This requires knowing the calorific density of each kind of food. The standard source for this is the USDA National Nutrient Database (NNDB). The latest version (May 2015) is Standard Release 27 [35], which lists nutritional facts about 8618 basic foods. However, we have noted a discrepancy of up to 35% in the calorie contents between the USDA NNDB and the numbers quoted by Nasco for their food replicas.13 A more serious problem is that the NNDB focuses on “raw” foods, rather than cooked meals. For example, NNDB contains information such as the calorie content of a pound of beef, but this does not help us estimate the calorie content of a cooked burger. For prepared foods, it is better to use the USDA Food and Nutrient Database for Dietary Studies (FNDDS) [16], which is derived from NNDB.14 However, the calorie content can vary a lot depending on exactly how the food was prepared (e.g., grilling vs frying). Consequently, we do not yet have a broad coverage nutritional database for prepared foods, and therefore we have not yet been able to conduct an end-to-end test of our system outside of the restaurant setting.15 13 According to http://www.enasco.com/product/ WA29109HR, “nutrition data is provided by The Nutrition Company using their FoodWorks nutrient analysis software”. Recall that these food replicas are used to train professional nutritionists, so the information cards that accompany them are designed to be as accurate as possible. 14 For a detailed comparison of NDB-SR and FNDDS, see http://www.nutrientdataconf.org/PastConf/NDBC36/ W-3_Montville_NNDC2012.pdf. 15 Companies such as MyFitnessPal have developed much larger nutri- Figure 9. Screenshot of the mobile app. Note that it is running in airplane mode (no internet connection). 8. Mobile app The complete system is sketched in Figure 1. We have ported the restaurant detector, meal detector, and food detectors (the multi-label classifiers) to the Android mobile phone system. The app can classify any image in under 1 second. The memory footprint for the model is less than 40MB (using unquantized floats to represent the CNN weights). However, we have not yet ported the segmentation or depth estimation CNNs. To use the app, the user takes a photo and then our system processes it. First the system determines if the image contains a meal, and if you are at a known restaurant. If so, it applies the multilabel classifiers. We sort the possible labels by their predicted probability, taking all those above a threshold of 0.5, and then truncating to the top 5, if necessary. We then show these labels to the user (see Figure 9 for a screenshot). The user can dismiss any labels that are incorrect. He/ she can also enter new labels, either by selecting from a pull-down menu (sorted in order of probability), or by entering free text. The user’s image and labels are then optionally stored in the cloud for subsequent offline model retraining (although we have not yet implemented this). 9. Related work There is a large number of related prior publications. Here we mention a few of them. In terms of public datasets, we have already mentioned Food101 [3], which has 101 classes and 101k images. In addition, there are various smaller datasets, such as: PFID [7], which has 61 classes (of fast food) and 1098 images; UNICT-FD889 [14], which has 889 classes and 3853 images; and UECFOOD-100 [24], which has 100 classes (of Japanese food), and 9060 images. tional databases using crowd sourcing [22], but this data is proprietary, and its quality is not guaranteed (since most casual users will not know the true nutritional content of their food). In terms of classifiers, [3, 18] use SVMs for generic foods. [2] and [1] develop restaurant-specific multi-label classifiers. Some recent papers on food segmentation include [28, 33, 38, 37]. There are many papers that leverage structure from motion to develop a 3d model of the food, including [28, 21, 10, 36, 32]. However, all these methods require multiple images and calibration markers. In terms of single images, [4] use parametric shape models for a small set of food types (e.g., sphere for an apple), and [19] use a nearest neighbor regression method to map the 2d image area to physical mass of each food item. There are very few papers that predict calories from images. [33] apply semantic image segmentation, and then train a support vector regression to map from the number of pixels of each class to the overall number of calories in the meal. [4, 19] calculate calories by multiplying the estimated volume of each food by the calorie density. [26] use crowdsourcing to estimate calories. [1] relies on the restaurant’s menu having the calorie information. 10. Discussion Besides its obvious practical use, the Im2Calories problem is also very interesting from the point of view of computer vision research. In particular, it requires solving various problems, such as: fine-grained recognition (to distinguish subtly different forms of food); hierarchical label spaces (to handle related labels); open-world recognition (to handle an unbounded number of class names); visual attribute recognition (to distinguish fried vs baked, or with or without mayo); instance segmentation; instance counting; amodal completion of occluded shapes [20]; depth estimation from a single image; information fusion from multiple images in real time, on-device; etc. We have tackled some of these problems, but it is clear that there is much more work to do. Nevertheless, we believe that even a partial solution to these problems could be of great value to people. References [1] O. Beijbom, N. Joshi, D. Morris, S. Saponas, and S. Khullar. Menu-match: restaurant-specific food logging from images. In WACV, 2015. [2] V. Bettadapura, E. Thomaz, A. Parnami, G. D. Abowd, and I. Essa. Leveraging context to support automated food recognition in restaurants. In WACV, pages 580–587, 2015. [3] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101: Mining discriminative components with random forests. In ECCV, 2014. [4] J. Chae, I. Woo, S. Kim, R. Maciejewski, F. Zhu, E. J. Delp, C. J. Boushey, and D. S. Ebert. Volume estimation using food specific shape templates in mobile image-based dietary assessment. In Proc. SPIE, 2011. [5] C. Champagne, G. Bray, A. Kurtz, J. Montiero, E. Tucker, J. Voaufova, and J. Delany. Energy intake and energy expenditure: a controlled study comparing dietitians and nondietitians. J. Am. Diet. Assoc., 2002. [6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected CRFs. In ICLR, 2015. [7] M. Chen, K. Dhingra, W. Wu, L. Yang, R. Sukthankar, and J. Yang. PFID: Pittsburgh fast-food image dataset. In ICIP, pages 289–292, 2009. [8] F. Cordeiro, E. Bales, E. Cherry, and J. Fogarty. Rethinking the mobile food journal: Exploring opportunities for lightweight Photo-Based capture. In CHI, 2015. [9] F. Cordeiro, D. Epstein, E. Thomaz, E. Bales, A. K. Jagannathan, G. D. Abowd, and J. Fogarty. Barriers and negative nudges: Exploring challenges in food journaling. In CHI, 2015. [10] J. Dehais, S. Shevchik, P. Diem, and S. G. Mougiakakou. Food volume computation for self dietary assessment applications. In 13th IEEE Conf. on Bioinfo. and Bioeng., pages 1–4, Nov. 2013. [11] J. Deng, J. Krause, A. C. Berg, and L. Fei-Fei. Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In CVPR, pages 3450–3457, June 2012. [12] D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common Multi-Scale convolutional architecture. Arxiv, 18 Nov. 2014. [13] M. Everingham, S. M. Ali Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111(1):98– 136, 25 June 2014. [14] G. M. Farinella, D. Allegra, and F. Stanco. A benchmark dataset to study the representation of food images. In ECCV Workshop Assistive CV, 2014. [15] FDA. www.fda.gov/Food/ IngredientsPackagingLabeling/LabelingNutrition/ucm248732.htm. [16] USDA FNDDS. www.ars.usda.gov/ba/bhnrc/fsrg. [17] B. Hariharan, P. Arbel, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV, 2014. [18] H. He, F. Kong, and J. Tan. DietCam: Multiview Food Recognition Using a MultiKernel SVM. IEEE J. of Biomedical and Health Informatics, 2015. [19] Y. He, C. Xu, N. Khanna, C. J. Boushey, and E. J. Delp. Food image analysis: Segmentation, identification and weight estimation. In ICME, pages 1–6, July 2013. [20] A. Kar, S. Tulsiani, J. Carreira, and J. Malik. Amodal completion and size constancy in natural scenes. In Intl. Conf. on Computer Vision, 2015. [21] F. Kong and J. Tan. DietCam: Automatic dietary assessment with mobile camera phones. Pervasive Mob. Comput., 8(1):147–163, Feb. 2012. [22] R. Macmanus. How myfitnesspal became the king of diet trackers, 2015. readwrite.com/2015/02/23/trackersmyfitnesspal-excerpt. [23] C. K. Martin, H. Han, S. M. Coulon, H. R. Allen, C. M. Champagne, and S. D. Anton. A novel method to remotely [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] measure food intake of free-living individuals in real time: the remote food photography method. British J. of Nutrition, 101(03):446–456, 2009. Y. Matsuda, H. Hoashi, and K. Yanai. Recognition of Multiple-Food images by detecting candidate regions. In ICME, pages 25–30, July 2012. Mealsnap app. tracker.dailyburn.com/apps. J. Noronha, E. Hysen, H. Zhang, and K. Z. Gajos. PlateMate: Crowdsourcing nutrition analysis from food photographs. In Proc. Symp. User interface software and tech., 2011. Google places API. https://developers.google.com/places/. M. Puri, Z. Zhu, Q. Yu, A. Divakaran, and H. Sawhney. Recognition and volume estimation of food intake using a mobile device. In WACV, pages 1–8, Dec. 2009. Rise app. https://www.rise.us/. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575, 2014. D. Schoeller, L. Bandini, and W. Dietz. Inaccuracies in selfreported intake identified by comparison with the doubly labelled water method. Can. J. Physiol. Pharm., 1990. T. Stutz, R. Dinic, M. Domhardt, and S. Ginzinger. Can mobile augmented reality systems assist in portion estimation? a user study. In Intl. Symp. Mixed and Aug. Reality, pages 51–57, 2014. K. Sudo, K. Murasaki, J. Shimamura, and Y. Taniguchi. Estimating nutritional value from food images based on semantic segmentation. In CEA workshop, pages 571–576, 13 Sept. 2014. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. USDA National Nutrient Database for Standard Reference, Release 27 (revised). http://www.ars.usda.gov/ba/bhnrc/ndl. C. Xu, N. Khanna, C. B. Y. He, and A. Parra. Image-Based food volume estimation. In CAE workshop, 2013. W. Zhang, Q. Yu, B. Siddiquie, A. Divakaran, and H. Sawhney. ’Snap-n-eat’: Food recognition and nutrition estimation on a smartphone. J. Diabetes Science and Technology, 9(3):525–533, 2015. F. Zhu, M. Bosch, N. Khanna, C. J. Boushey, and E. J. Delp. Multiple hypotheses image segmentation and classification with application to dietary assessment. IEEE J. of Biomedical and Health Informatics, 19(1):377–388, Jan. 2015. F. Zhu, M. Bosch, I. Woo, S. Kim, C. J. Boushey, D. S. Ebert, and E. J. Delp. The use of mobile devices in aiding dietary assessment and evaluation. IEEE J. Sel. Top. Signal Process., 4(4):756–766, Aug. 2010. DeepBox: Learning Objectness with Convolutional Networks Weicheng Kuo Bharath Hariharan Jitendra Malik University of California, Berkeley {wckuo, bharath2, malik}@eecs.berkeley.edu Abstract and let the algorithm figure out what low-, mid- and highlevel cues are most discriminative of objects. Following recent work on a range of recognition tasks [11, 12, 18], we use convolutional networks (CNNs) [19] for this task. Concretely, we train a CNN to rerank a large pool of object proposals produced by a bottom-up proposal method (we use Edge boxes [30] for most experiments in this paper). For ease of reference, we call our approach DeepBox. Figure 1 shows our framework. We propose a lightweight four layer network architecture that significantly improves over bottom-up proposal methods in terms of ranking (26% relative improvement on AUC over Edge boxes on VOC 2007 [8]). Our network architecture is as effective as state-of-the-art classification networks on this task while being much smaller and thus much faster. In addition, using ideas from SPP [13] and Fast RCNN [10], our implementation runs in 260 ms per image, comparable to some of the fastest bottom-up proposal approaches like Edge Boxes (250 ms). We also provide evidence that what our network learns is category-agnostic: our improvements in performance generalize to categories that the CNN has not seen before (16% improvement over Edge boxes on COCO [20]). Our results suggest that a) there is indeed a generic, semantic notion of objectness beyond bottom-up saliency, and that b) this semantic notion of objectness can be learnt effectively by a lightweight CNN. Object proposals are just the first step in an object detection system, and the final evaluation of a proposal system is the impact it has on detection performance. We show that the Fast R-CNN detection system [10], using 500 DeepBox proposals per image, is 4.5 points better than the same object detector using 500 Edge box proposals. Thus our high quality proposals directly lead to better object detection. The rest of the paper is laid out as follows. In Section 2 we discuss related work. We describe our network architecture and training and testing procedures in Section 3. Section 4 describes experiments and we end with a discussion. Existing object proposal approaches use primarily bottom-up cues to rank proposals, while we believe that “objectness” is in fact a high level construct. We argue for a data-driven, semantic approach for ranking object proposals. Our framework, which we call DeepBox, uses convolutional neural networks (CNNs) to rerank proposals from a bottom-up method. We use a novel four-layer CNN architecture that is as good as much larger networks on the task of evaluating objectness while being much faster. We show that DeepBox significantly improves over the bottomup ranking, achieving the same recall with 500 proposals as achieved by bottom-up methods with 2000. This improvement generalizes to categories the CNN has never seen before and leads to a 4.5-point gain in detection mAP. Our implementation achieves this performance while running at 260 ms per image. 1. Introduction Object detection methods have moved from scanning window approaches [9] to ones based on bottom-up object proposals [11]. Bottom-up proposals [1] have two major advantages: 1) by reducing the search space, they allow the usage of more sophisticated recognition machinery, and 2) by pruning away false positives, they make detection easier. [14, 10] Most object proposal methods rely on simple bottom-up grouping and saliency cues. The rationale for this is that this step should be reasonably fast and generally applicable to all object categories. However, we believe that there is more to objectness than bottom-up grouping or saliency. For instance, many disparate object categories might share highlevel structures (such as the limbs of animals and robots) and detecting such structures might hint towards the presence of objects. A proposal method that incorporates these and other such cues is likely to perform much better. In this paper, we argue for a semantic, data-driven notion of objectness. Our approach is to present a large database of images with annotated objects to a learning algorithm, 2. Related work Russell et al. [26] were one of the first to suggest a category-independent method to propose putative objects. 1 3. Method Figure 1. The DeepBox framework. Given any RGB image, we first generate bottom-up proposals and then rerank them using a CNN. High ranked boxes are shown in green and low ranked ones are blue. The number in each box is its ranking in the proposal pool. DeepBox corrects the ranking of Edge box, ranking objects higher than background. Their method involved sampling regions from multiple segmentations of the image. More recently, Alexe et al. [1] and Endres et al. [6] propose using bottom-up object proposals as a first step in recognition. Expanding on the multiple segmentations idea, Selective search [29] uses regions from hierarchical segmentations in multiple color spaces as object proposals. CPMC [4] uses multiple graph-cut based segmentations with multiple foreground seeds and multiple foreground biases to propose objects. GOP [17] replaces graph cuts with a much faster geodesic based segmentation. MCG [2] also uses multiple hierarchical segmentations from different scales of the image, but produces proposals by combinatorially grouping regions. Edge boxes [30] uses contour information instead of segments: bounding boxes which have fewer contours straggling the boundary of the box are considered more likely to be objects. Many object proposal methods also include a ranking of the regions. This ranking is typically based on low level region features such as saliency [1], and is sometimes learnt [2, 4]. Relatively simple ranking suffices when the goal is a few thousand proposals as in MCG [2], but to narrow the list down to a few hundred as in CPMC [4] requires more involved reasoning. DeepBox aims at such a ranking. Multibox [7, 28] directly produces object proposals from images using a sophisticated neural network. In contemporary work, Faster R-CNN [25] uses the same large network to propose objects and classify them. DeepMask [22] also uses a very deep network to directly produce segment proposals. In comparison, our architecture is quite lightweight and can be used out of the box to rerank any bottom-up proposals. Finally, we direct the reader to [15, 14] for a more thorough evaluation of bottom-up proposal methods. The pipeline consists of two steps: 1) Generate an initial pool of N bottom-up proposals. Our method is agnostic to the precise bottom-up proposal method. The main point of this step is to prune out the obviously unlikely windows so that DeepBox can focus on the hard negatives. 2) Rerank the proposals using scores obtained by the DeepBox network. We rerank each proposal by cropping out the proposal box and feeding it into a CNN, as described by Girshick et al. [11]. Because highly overlapping proposals are handled independently, this strategy is computationally wasteful and thus slow. A more sophisticated and much faster approach, using ideas from [13, 10] is described in Section 3.2. For datasets without many small objects (e.g. PASCAL), we often only need to re-rank the top 2000 proposals to obtain good enough recall. As shown by [30], increasing the number of Edge box proposals beyond 2000 leads to only marginal increase in recall. For more challenging datasets with small objects (e.g. COCO [20]), reranking more proposals continues to provide gains in recall beyond 2000 proposals. 3.1. Network Architecture When it comes to the network architecture, we would expect that predicting the precise category of an object is harder than predicting objectness, and so we would want a simpler network for objectness. This also makes sense from a computational standpoint, since we do not want the object proposal scoring stage to be as expensive as the detector itself. We organized our search for a suitable network architecture by starting from the architecture of [18] and gradually ablating it while trying to preserve performance. The ablation here can be performed by reducing the number of channels in the different layers (thus reducing the number of parameters in the layer), by removing some layers, or by decreasing the input resolution so that the features computed become coarser. The original architecture gave an AUC on PASCAL VOC of 0.76(0.62) for IoU=0.5 (0.7). First, we changed the number of outputs of f c6 to 1024 with other things fixed and found that performance remained unchanged. Then we adjusted the input image crop from 227 × 227 of the network to 120 × 120 and observed that the AUC dropped 1.5 points for IoU=0.5 and 2.9 points for IoU=0.7 on PASCAL. With this input size, we tried removing f c6 (drop: 2.3 points), conv5 (drop: 2.9 points), conv5 + conv4 (drop: 10.6 points) and conv5 + conv4 + conv3 layers (drop: 6.7 points). This last experiment meant that dropping all of conv5, conv4 and conv3 was better than just dropping conv5 and conv4. This might be because conv3, conv4 and conv5 while adding to the capacity of the network are also likely overfit to the task of image classification (as described below, the convolutional layers are initialized from a model trained on ImageNet). We stuck to this architecture (i.e.,without conv5, conv4 and conv3) and explored different input sizes for the net. For an input size of 140 × 140, we obtained a competitive AUC of 0.74 (for IoU=0.5) and 0.60 (for IoU=0.7) on PASCAL, or equivalently a 4.4 point drop against the baseline. Our final architecture can be written down as follows. Denote by conv(k, c, s) a convolutional layer with kernel size k, stride s and number of output channels c. Similarly, pool(k, s) denotes a pooling layer with kernel size k and stride s, and f c(c) a fully connected layer with c outputs. Then, our network architecture is: conv(11, 96, 4)−pool(3, 2)−conv(5, 256, 1)−f c(1024)− f c(2). Each layer except the last is followed by a ReLU nonlinearity. Our problem is a binary classification problem (object or not), so we only have two outputs which are passed through a softmax. Our input size is 140 × 140. While we finalized this architecture on PASCAL for Edge boxes, we show in Section 4 that the same architecture works just as well for other datasets such as COCO [20] and for other proposal methods such as Selective Search [29] or MCG [2]. 3.2. Sharing computation for faster reranking Running the CNN separately on highly overlapping boxes wastes a lot of computation. He et al. [13] pointed out that the convolutional part of the network could be shared among all the proposals. Concretely, instead of cropping out individual boxes, we pass in the entire image into the network (at a high resolution). After passing through all the convolutional and pooling layers, the result is a feature map which is some fraction of the image size. Given this feature map and a set of bounding boxes, we want to compute a fixed length vector for each box that we can then feed into the fully connected layers. To do this, note that each bounding box B in the image space corresponds to a box b in the feature space of the final convolutional layer, with the scale and aspect ratio of b being dependent on B and thus different for each box. He et al. propose to use a fixed spatial pyramid grid to max-pool features for each box. While the size of the grid cell varies with the box, the number of bins don’t, thus resulting in a fixed feature vector for each box. From here on, each box is handled separately. However, the shared convolutional feature maps means that we have saved a lot of computation. One issue with this approach is that all the convolutional feature maps are computed at just one image scale, which may not be appropriate for all objects. He et al. [13] suggest a multiscale version where the feature maps are computed at a few fixed scales, and for each box we pick the best scale, which they define as the one where the area of the scaled box is closest to a predefined value. We experiment with both the single scale version and a multiscale version using three scales. We use the implementation proposed by Girshick in Fast R-CNN [10]. Fast R-CNN implements this pooling as a layer in the CNN (the RoI Pooling layer) allowing us to train the network end-to-end. To differentiate this version of DeepBox from the slower alternative based on cropping and warping, we call this version Fast DeepBox in the rest of the paper. 3.3. Training Procedure 3.3.1 Initialization The first two convolutional layers were initialized using the publicly available Imagenet model [18]. This model was pretrained on 1000 Imagenet categories for the classification task. The f c layers are initialized randomly from Gaussian distribution with σ = 0.01. Our DeepBox training procedure consists of two stages. Similar to the classical notion of bootstrapping in object detection, we first train an initial model to distinguish between object boxes and randomly sampled sliding windows from the background. This teaches it the difference between objects and background. To enable it to do better at correcting the errors made by bottom-up proposal methods, we run a second training round where we train the model on bottom-up proposals from a method such as Edge boxes. 3.3.2 Training on Sliding Windows First we generate negative windows by simple raster scanning. The sliding window step size is selected based on the box-searching strategy of Edge boxes[30]. Following Zitnick et al [30], we use α to denote the IoU threshold for neighboring sliding windows, and set it to 0.65. We generated windows in 5 aspect ratios: (w : h) = (1 : 1), (2 : 3), (1 : 3), (3 : 2), and (3 : 1). Negative windows which overlap with a ground truth object by more than β− = 0.5 are discarded. To obtain positives, we randomly perturb the corners of ground truth bounding boxes. Suppose a ground truth bounding box has coordinates (xmin , ymin , xmax , ymax ), with the width denoted by w and the height denoted by h. Then the perturbed coordinates are distributed as: x0min ∼ unif(xmin − γw, xmin + γw) (1) 0 ymin 0 xmax 0 ymax ∼ unif(ymin − γh, ymin + γh) (2) ∼ unif(xmax − γw, xmax + γw) (3) ∼ unif(ymax − γh, ymax + γh) (4) where γ = 0.2 defines the level of noise. Larger γ introduces more robustness into positive training samples, but might hurt localization. In practice we found that γ = 0.2 works well. In case some perturbed points go out of the image, we set them to stay on the image border. Perturbed windows that overlap with ground truth boxes by less than β+ = 0.5 are discarded. 3.3.3 Training on Hard-negatives Next, we trained the net on bottom-up proposals from a method such as Edge boxes [30]. Compared to sliding windows, these proposals contain more edges, texture and parts of objects, and thus form hard negatives. While sliding windows trained the net to distinguish primarily between background and objects, training on these proposals allows the net to learn the notion of complete objects and better adapt itself to the errors made by the bottom-up proposer. This is critical: without this stage, performance drops by 18% on PASCAL from 0.60 AUC at IoU=0.7 to 0.48. The window preparation and training procedure is as described in Section 3.3.2, except that sliding windows are replaced with Edge boxes proposals. We set the overlap threshold β+ = 0.7 slightly higher, so that the net can learn to distinguish between tight and loose bounding boxes. We set β− = 0.3 below which the windows are labeled negative. Windows with IoU ∈ [0.3, 0.7] are discarded lest they confuse the net. We balanced the ratio of positive and negative windows at 1 : 3 at training time for both stages. Momentum is set to 0.9 and weight decay to 0.0005. The initial learning rate is 0.001, which we decrease by 0.1 every 20,000 iterations. We train DeepBox for 60,000 iterations with a mini-batch size of 128. The Fast DeepBox network was trained for 120k iterations in both sliding window and hard negative training. Three scales [400, 600, 900] were used for both training and testing. 4. Experiments All experiments except those in Section 4.7 and 4.8 were done using DeepBox. We evaluate Fast DeepBox in Section 4.7, and as a final experiment plug it into an object detection system in Section 4.8. 4.1. Learning objectness on PASCAL and COCO In our first set of experiments, we evaluated our approach on PASCAL VOC 2007 [8] and on the newly released COCO [20] dataset. For these experiments, we used Edge boxes [30] for the initial pool of proposals. On PASCAL we reranked the top 2048 Edge box proposals, whereas on COCO we re-ranked all. We used the network architecture described in Section 3.1. For results on PASCAL VOC 2007, we trained our network on the trainval set and tested on the test set. For results on COCO, we trained our network on the train set and evaluated on the val set. 4.2. Comparison to Edge boxes We first compare our ranking to the ranking output by Edge boxes. Figure 2 plots Recall vs Number of Proposals in PASCAL VOC 2007 for IoU=0.71 . We observe that DeepBox outperforms Edge boxes in all regimes, especially with a low number of proposals. The same is true for IoU=0.5 (not shown). The AUCs (Areas Under Curve) for DeepBox are 0.74(0.60) vs Edge box’s 0.60(0.47) for IoU=0.5(0.7), suggesting that DeepBox proposals are 24% better at IoU=0.5 and 26% better at IoU=0.7 compared to Edge boxes. Figure 3 plots the same in COCO. The AUCs (Areas Under Curve) for DeepBox are 0.40(0.28) vs Edge boxes’s 0.28(0.20) for IoU=0.5(0.7), suggesting DeepBox proposals are 40%(43%) better than Edge boxes. If we re-ranked top 2048 proposals instead, the AUCs are 0.38(0.27). On PASCAL, we also plot the Recall vs IoU threshold curves at 1000 proposals in Figure 4. At 1000 proposals, the gain of DeepBox is not as big as at 100-500 proposals, but we still see that it is superior in all regimes of IoU. Part of this performance gain is due to small objects, defined as objects with area less than 2000. On COCO the AUCs for DeepBox (trained on all categories) are (0.161, 0.080), while the numbers for Edge boxes are (0.061, 0.030) for IoU(0.5,0.7). DeepBox outperforms Edge boxes by more than 160% on small objects. Comparison to other proposal methods We can also compare our ranked set of proposals to all the other proposals in the literature. We show this comparison for IoU=0.7 in Table 1. The numbers are obtained from [30] except for MCG and DeepBox which we computed ourselves. In Table 1, the metrics are the number of proposals needed to achieve 25%, 50% and 75% recall and the maximum recall using 5000 boxes. 4.3. Visualization of DeepBox proposals Figures 5 and 6 visualizes DeepBox and Edge box performance on PASCAL and COCO images. Figure 5 shows the ground truth boxes that are detected (“hits”, shown in green, with the corresponding best overlapping proposal shown in blue) and those that are missed (shown in red) for both proposal rankings. We observe that in complicated scenes with multiple small objects or cluttered background, DeepBox significantly outperforms Edge box. The tiny boats, the cars parked by the road, the donuts and people in the shade are all correctly captured. 1 We computed recall using the code provided by [30] Figure 5. Visualization of hits and misses. In each image, the green boxes are ground truth boxes for which a highly overlapping proposal exists (with the corresponding proposals shown as blue boxes) and red boxes are ground truth boxes that are missed. The IoU threshold is 0.7. We evaluated 1000 proposals per image for COCO and 500 proposals per image for PASCAL. The left image in every pair shows the result of DeepBox ranking, and the right image shows the ranking from Edge boxes. In cluttered scenes, DeepBox has a higher recall. See Section 4.3 for a detailed discussion. Figure 6. In each image we show the distribution of the proposals produced by pasting a red box for each proposal. Only the top 100 proposals are shown. For each pair of images, DeepBox is on the left and Edge boxes is the right. DeepBox proposals are more tightly concentrated on the objects. See Section 4.3 for a detailed discussion. This is also validated by looking at the distribution of top 100 proposals (Figure 6), which is shown in red. In general, DeepBox’s bounding boxes are very densely focused on the objects of interest while Edge boxes primarily recognize contours and often spread evenly across the image in a complicated scene. 4.4. Generalization to unseen categories The high recall our method achieves on the PASCAL 2007 test set does not guarantee that our net is truly learning PASCAL Evaluation IoU=0.7 AUC 25% 50% 75% 0.20 0.23 0.27 0.35 0.37 0.40 0.41 0.48 0.47 0.60 292 184 27 42 29 28 15 10 12 3 584 349 307 199 111 81 108 20 3023 1434 871 800 183 Recall (%) 29 68 39 80 70 87 65 83 87 87 1 0.9 DeepBox Edge boxes BING[5] Ranta[24] Objectness[1] Rand. P.[21] Rahtu [23] SelSearch [29] CPMC [3] MCG [2] E.B [30] DeepBox 0.8 0.7 Recall 0.6 0.5 0.4 0.3 0.2 0.1 0 0 10 1 2 10 10 Number of proposals 3 10 Figure 2. PASCAL Evaluation IoU=0.7. DeepBox starts off much higher than Edge boxes. The wide margin continues all the way until 500 proposals and gradually decays. The two curves join at 2048 proposals because we chose to re-rank this number of proposals. Table 1. Comparison of DeepBox with existing object proposal techniques. All numbers are for an IoU threshold of 0.7. The metrics are the number of proposals needed to achieve 25%, 50% and 75% recall and the maximum recall using 5000 boxes. DeepBox outperforms in all metrics. Note that the timing numbers for DeepBox include computation on the GPU. PASCAL Evaluation 1000 proposals 1 0.9 DeepBox Edge boxes 0.8 COCO Evaluation IoU=0.7 0.7 DeepBox Edge boxes 0.6 Recall 0.5 0.5 0.4 0.4 Recall Time (s) 0.2 10 3 1 3 10 250 34 0.25 2.5 0.3 0.3 0.2 0.1 0.2 0 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 IoU threshold 0.1 0 0 10 1 10 2 10 Number of proposals 3 10 Figure 3. MS COCO Evaluation IoU=0.7. The strong gain demonstrated by DeepBox on COCO suggests that our learnt objectness is particularly helpful in a complicated dataset. general objectness. It is possible that the net is learning just the union of 20 object categories and using that knowledge to rank proposals. To evaluate whether the net is indeed learning a more general notion of objectness that extends beyond the categories it has seen during training, we did the following experiment: • We identified the 36 overlapping categories between Imagenet and COCO. • We trained the net just on these overlapping categories on COCO with initialization from Imagenet. This 1 Figure 4. Variation of recall with IoU threshold at 1000 proposals. DeepBox (average recall: 57.0%) is better than Edge boxes (average recall: 54.4%) in all regimes. Comparisons to other proposal methods is shown in the supplementary. means that during training, only boxes that overlapped highly with ground truth from the 36 overlapping categories were labeled positives, and others were labeled negatives. Also, when sampling positives, only the ground truth boxes corresponding to these overlapping categories were used to produce perturbed positives. This is equivalent to training on a dataset where all the other categories have not been labeled at all. • We then evaluated our performance on the rest of the categories in COCO (44 in number). This means that at test time, only proposed boxes that overlapped with ground truth from the other 44 categories were considered true positives. Again, this corresponds to evaluat- Vanilla COCO Evaluation IoU=0.5 0.7 DeepBox Edge boxes Sel. Se. MCG Edge Box 0.6 Recall 0.5 0.27/0.17 0.38/0.25 0.28/0.20 DeepBox (Trained on [30]) 0.27/0.19 0.30/0.22 0.38/0.27 DeepBox (Finetuned) 0.32/0.22 0.37/0.26 0.38/0.27 Total Time (s) 12.5 36.5 2.5 Table 2. DeepBox on top of other proposers. For each method we show the AUC at IoU=0.5/0.7 of (left to right) the original ranking, the reranking produced by DeepBox trained on Edge boxes, and that produced after finetuning on the corresponding proposals. 0.4 0.3 0.2 0.1 0 0 10 4.5. DeepBox using Large Networks 1 10 2 10 Number of proposals 3 10 Figure 7. Evaluation on unseen categories, when ranking all proposals, at IoU=0.5. ing on a dataset where the 36 training categories have not been labeled at all. All other experimental settings remain the same. As before, we use Edge boxes for the initial pool of proposals which we rerank, and compare the ranking we obtain to the ranking output by edge boxes. When reranking the top 2048 proposals, DeepBox achieved 15.7%(15.3%) AUC improvement over Edge boxes for IoU=0.5 (0.7). When reranking all proposals output by Edge boxes, DeepBox achieved 18.5%(16.2%) improvement over Edge boxes for IoU=0.5 (0.7) (Figure 7). In this setting, DeepBox outperforms Edge box in all regimes on unseen categories for both IoUs. In Figure 8, we plot the ratio of the AUC obtained by DeepBox to that obtained by Edge box for all the 44 testing categories. In more than half of the testing categories, we obtain an improvement greater than 20%, suggesting that the gains provided by DeepBox are spread over multiple categories. This suggests that DeepBox is actually learning a class-agnostic notion of objectness. Note that DeepBox performs especially well for the animal super category because all animal categories have similar geometric structure and training the net on some animals helps it recognize other animals. This validates our intuition that there are high-level semantic cues that can help with objectness. In the sports super category, DeepBox performs worse than Edge boxes, perhaps because most objects of this category have salient contours that favor the Edge boxes algorithm. These results on COCO demonstrate that our network has learnt a notion of objectness that generalizes beyond training categories. We also experimented with larger networks such as “Alexnet” [18] and “VGG” [27]. Unsurprisingly, large networks capture objectness better than our small network. However, the difference is quite small: With VGG, the AUCs on PASCAL are 0.78(0.65) for IoU=0.5(0.7). The numbers for Alexnet are 0.76(0.62). In comparison our network achieves 0.74(0.60). For evaluation on COCO, we randomly selected 5000 images and computed AUCs using the VGG and Alexnet architecture. All networks re-rank just top 2048 Edge box proposals. At IoU=0.5, VGG gets 0.43 and Alexnet gets 0.42, compared to the 0.38 obtained by our network. At IoU=0.7, VGG gets 0.31 and Alexnet gets 0.30, compared to the 0.27 obtained by our network. When re-ranking all proposals, our small net gets 0.40(0.28) for IoU=0.5(0.7). These experiments suggest that our network architecture is sufficient to capture the semantics of objectness, while also being much more efficient to evaluate compared to the more massive VGG and Alexnet architectures. 4.6. DeepBox with Other Proposers It is a natural question to ask whether DeepBox framework applies to other bottom-up proposers as well. We experimented with MCG and Selective Search by just reranking top 2048 proposals. (For computational reasons, we did this experiment on a smaller set of 5000 COCO images.) We experimented with two kinds of DeepBox models: a single model trained on Edge boxes, and separate models trained on each proposal method (Table 2). A model trained on Edge boxes does not provide much gains, and indeed sometimes hurts, when run on top of Selective Search or MCG proposals. However, if we retrain DeepBox separately on each proposal method, this effect goes away. In particular, we get large gains on Selective Search for both IoU thresholds (as with Edge boxes). On MCG, DeepBox does not hurt, but does not help much either. Nevertheless, the gains we get on Edge boxes and Selective Search suggest that our approach is general and can work with any proposal method, or even ensembles of proposal methods (a possibility we leave for future work). Figure 8. Evaluation on unseen categories: category-wise breakdown. This demonstrates that DeepBox network has learnt a notion of objectness that generalizes beyond training categories. 4.7. Fast DeepBox We experimented with Fast DeepBox on COCO. The AUCs for multiscale Fast DeepBox are 0.40(0.29) vs Edge box’s 0.28(0.20) for IoU=0.5(0.7), a gain of 41%(44%) over Edge boxes. When we re-ranked top 2048 proposals instead, the AUCs are 0.37(0.27). Compared with DeepBox’s multi-thread runtime of 2.5s, Fast DeepBox (multiscale) is an order of magnitude faster: it takes 0.26s to rerank all proposals or 0.12s to re-rank the top-2000. This compares favorably to other bottom-up proposals. In terms of average recall with 1000 proposals, our performance (0.39) is better than GOP (0.36) [17], Selective Search (0.36) [29] and Edge boxes (0.34) [30], and is about the same as MCG (0.40) [2] while being almost 70 times faster. In contrast, Deepmask achieves 0.45 with a much deeper network at the expense of being 3 times slower (1.6 s) [22]. With a small decrease in performance, Fast DeepBox can be made much faster. With a single scale, AUC drops by about 0.005 when re-ranking top-2000 proposals and 0.01 when re-ranking all proposals. However, it only takes 0.11s to re-rank all proposals or 0.060s for the top-2000. One can also make training faster by removing the scanning-window stage and using a single scale. This speedup comes with a drop in performance of 0.015 when reranking all proposals compared to the multiscale two-stage version. 4.8. Impact on Object Detection The final metric for any proposal method is its impact on object detection. Good proposals not only reduce the computational complexity but can also make object detection easier by reducing the number of candidates that the detector has to choose from [14, 10]. We found that this is indeed true: when using 500 DeepBox proposals, FastRCNN (with the VGG-16 network) gives a mAP of 37.8% on COCO Test at IoU=0.5, compared to only 33.3% when using 500 Edge Box proposals. Even when using 2000 Edge Box proposals, the mAP is still lower (35.9%). For comparison, Fast R-CNN using 2000 Selective Search proposals gets a mean AP of 35.8%, indicating that with just 500 DeepBox proposals we get a 2 point jump in performance. 5. Discussion and Conclusion We have presented an efficient CNN architecture that learns a semantic notion of objectness that generalizes to unseen categories. We conclude by discussing other applications of our objectness model. First, as the number of object categories increases, the computational complexity of the detector increases and it becomes more and more useful to have a generic objectness system to reduce the number of locations the detector looks at. Objectness can also help take the burden of localization off the detector, which then has an easier task. Second, AI agents navigating the world cannot expect to be trained on labeled data like COCO for every object category they see. For some categories the agent will have to collect data and build detectors on the fly. In this case, objectness allows the agent to pick a few candidate locations in a scene that look like objects and track them over time, thus collecting data for training a detector. Objectness can thus be useful for object discovery [16], especially when it captures semantic properties as in our approach. 6. Acknowledgement This work is supported by a Berkeley Graduate Fellowship and a Microsoft Research Fellowship. We thank NVIDIA for giving GPUs through their academic program too. References [1] B. Alexe, T. Deselaers, and V. Ferrari. Measuring the objectness of image windows. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(11):2189–2202, 2012. 1, 2, 6 [2] P. Arbeláez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale combinatorial grouping. In CVPR, 2014. 2, 3, 6, 8 [3] J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu. Semantic segmentation with second-order pooling. In ECCV, 2012. 6 [4] J. Carreira and C. Sminchisescu. Cpmc: Automatic object segmentation using constrained parametric min-cuts. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(7):1312–1328, 2012. 2 [5] M. Cheng, Z. Zhang, W. Lin, , and P. Torr. Bing: Binarized normed gradients for objectness estimation at 300fps. In CVPR, 2014. 6 [6] I. Endres and D. Hoiem. Category independent object proposals. In Computer Vision–ECCV 2010, pages 575–588. Springer, 2010. 2 [7] D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov. Scalable object detection using deep neural networks. In CVPR, 2014. 2 [8] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010. 1, 4 [9] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 32(9), 2010. 1 [10] R. Girshick. Fast R-CNN. In ICCV, 2015. 1, 2, 3, 8 [11] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 1, 2 [12] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV, 2014. 1 [13] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV, 2014. 1, 2, 3 [14] J. Hosang, R. Benenson, P. Dollár, and B. Schiele. What makes for effective detection proposals? arXiv preprint arXiv:1502.05082, 2015. 1, 2, 8 [15] J. Hosang, R. Benenson, and B. Schiele. How good are detection proposals, really? In BMVC, 2014. 2 [16] H. Kang, M. Hebert, A. A. Efros, and T. Kanade. Connecting missing links: Object discovery from sparse observations using 5 million product images. ECCV. 2012. 8 In [17] P. Krähenbühl and V. Koltun. Geodesic object proposals. In ECCV, 2014. 2, 8 [18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 1, 2, 3, 7 [19] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989. 1 [20] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollr, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 1, 2, 3, 4 [21] S. Manen, M. Guillaumin, and L. Van Gool. Prime object proposals with randomized prim’s algorithm. In ICCV, 2013. 6 [22] P. O. Pinheiro, R. Collobert, and P. Dollár. Learning to segment object candidates. In NIPS, 2015 (To appear). 2, 8 [23] E. Rahtu, K. Juho, and B. Matthew. Learning a category independent object detection cascade. In ICCV, 2011. 6 [24] R. E. Rantalankila P., Kannala J. Generating object segmentation proposals using global and local search. In CVPR, 2014. 6 [25] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster RCNN: towards real-time object detection with region proposal networks. In NIPS, 2015 (To appear). 2 [26] B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman. Using multiple segmentations to discover objects and their extent in image collections. In CVPR, 2006. 1 [27] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 7 [28] C. Szegedy, S. Reed, D. Erhan, and D. Anguelov. Scalable, high-quality object detection. arXiv preprint arXiv:1412.1441, 2014. 2 [29] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. IJCV, 104(2), 2013. 2, 3, 6, 8 [30] C. L. Zitnick and P. Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014. 1, 2, 3, 4, 6, 7, 8 Flowing ConvNets for Human Pose Estimation in Videos Tomas Pfister Dept. of Engineering Science University of Oxford James Charles School of Computing University of Leeds Andrew Zisserman Dept. of Engineering Science University of Oxford tp@robots.ox.ac.uk j.charles@leeds.ac.uk az@robots.ox.ac.uk Abstract dense optical flow vectors to warp predicted positions onto a target image. In particular, we show that when regressing a heatmap of positions (in our application for human joints), the heatmaps from neighbouring frames can be warped and aligned using optical flow, effectively propagating position confidences temporally, as illustrated in Fig 1. We also propose a deeper architecture that has additional convolutional layers beyond the initial heatmaps to enable learning an implicit spatial model of human layout. These layers are able to learn dependencies between human body parts. We show that these ‘spatial fusion’ layers remove pose estimation failures that are kinematically impossible. The objective of this work is human pose estimation in videos, where multiple frames are available. We investigate a ConvNet architecture that is able to benefit from temporal context by combining information across the multiple frames using optical flow. To this end we propose a network architecture with the following novelties: (i) a deeper network than previously investigated for regressing heatmaps; (ii) spatial fusion layers that learn an implicit spatial model; (iii) optical flow is used to align heatmap predictions from neighbouring frames; and (iv) a final parametric pooling layer which learns to combine the aligned heatmaps into a pooled confidence map. We show that this architecture outperforms a number of others, including one that uses optical flow solely at the input layers, one that regresses joint coordinates directly, and one that predicts heatmaps without spatial fusion. The new architecture outperforms the state of the art by a large margin on three video pose estimation datasets, including the very challenging Poses in the Wild dataset, and outperforms other deep methods that don’t use a graphical model on the single-image FLIC benchmark (and also [5, 35] in the high precision region). Related work. Traditional methods for pose estimation have often used pictorial structure models [2, 8, 10, 27, 39], which optimise a configuration of parts as a function of local image evidence for a part, and a prior for the relative positions of parts in the human kinematic chain. An alternative approach uses poselets [1, 13]. More recent work has tackled pose estimation holistically: initially with Random Forests on depth data [12, 29, 31, 34] and RGB [3, 4, 24], and most recently with convolutional neural networks. The power of ConvNets has been demonstrated in a wide variety of vision tasks – object classification and detection [11, 21, 28, 40], face recognition [32], text recognition [15, 16], video action recognition [20, 30] and many more [7, 22, 25]. For pose estimation, there were early examples of using ConvNets for pose comparisons [33]. More recently, [37] used an AlexNet-like ConvNet to directly regress joint coordinates, with a cascade of ConvNet regressors to improve accuracy over a single pose regressor network. Chen and Yuille [5] combine a parts-based model with ConvNets (by using a ConvNet to learn conditional probabilities for the presence of parts and their spatial relationship with image patches). In a series of papers, Tompson, Jain et al. developed ConvNet architectures to directly regress heatmaps for each joint, with subsequent layers to add an Markov Random Field (MRF)-based spatial model [17, 36], and a pose refinement model [35] (based on a Siamese network with shared weights) upon a rougher pose estimator ConvNet. 1. Introduction Despite a long history of research, human pose estimation in videos remains a very challenging task in computer vision. Compared to still image pose estimation, the temporal component of videos provides an additional (and important) cue for recognition, as strong dependencies of pose positions exist between temporally close video frames. In this work we propose a new approach for using optical flow for part localisation in deep Convolutional Networks (ConvNets), and demonstrate its performance for human pose estimation in videos. The key insight is that, since for localisation the prediction targets are positions in the image space (e.g. (x, y) coordinates of joints), one can use 1 Input conv3 conv4 t conv5 ... conv6 conv7 conv8 t+n conv1_f Warped heatmaps + ... conv2 Optical flow Temporal Pooler conv2_f conv9 conv3_f 1x1x7 Pooled heatmap for frame t conv4_f conv5_f ... ... conv1 Spatial Fusion Layers ... SpatialNet ... t-n Pose heatmaps Loss 2 Loss 1 + Output Figure 1. Deep expert pooling architecture for pose estimation. The network takes as an input all RGB frames within a n-frame neighbourhood of the current frame t. The fully convolutional network (consisting of a heatmap net with an implicit spatial model) predicts a confidence heatmap for each body joint in these frames (shown here with a different colour per joint). These heatmaps are then temporally warped to the current frame t using optical flow. The warped heatmaps (from multiple frames) are then pooled with another convolutional layer (the temporal pooler), which learns how to weigh the warped heatmaps from nearby frames. The final body joints are selected as the maximum of the pooled heatmap (illustrated here with a skeleton overlaid on top of the person). Temporal information in videos was initially used with ConvNets for action recognition [30], where optical flow was used as an input motion feature to the network. Following this work, [18, 24] investigated the use of temporal information for pose estimation in a similar manner, by inputting flow or RGB from multiple nearby frames into the network, and predicting joint positions in the current frame. However, pose estimation differs from action recognition in a key respect which warrants a different approach to using optical flow: in action recognition the prediction target is a class label, whereas in pose estimation the target is a set of (x, y) positions onto the image. Since the targets are positions in the image space, one can use dense optical flow vectors not only as an input feature but also to warp predicted positions in the image, as done in [4] for random forest estimators. To this end, our work explicitly predicts joint positions for all neighbouring frames, and temporally aligns them to frame t by warping them backwards or forwards in time using tracks from dense optical flow. This effectively reinforces the confidence in frame t with a strong set of ‘expert opinions’ (with corresponding confidences) from neighbouring frames, from which joint positions can be more precisely estimated. Unlike [4] who average the expert opinions, we learn the expert pooling weights with backpropagation in an end-to-end ConvNet. Our ConvNet outperforms the state of the art on three challenging video pose estimation datasets (BBC Pose, ChaLearn and Poses in the Wild) – the heatmap regressor alone surpasses the state of the art on these datasets, and the pooling from neighbouring frames using optical flow gives a further significant boost. We have released the models and code at http://www.robots.ox.ac.uk/˜vgg. 2. Temporal Pose Estimation Networks Fig 1 shows an overview of the ConvNet architecture. Given a set of input frames within a temporal neighbourhood of n frames from a frame t, a spatial ConvNet regresses joint confidence maps (‘heatmaps’) for each input frame separately. These heatmaps are then individually warped to frame t using dense optical flow. The warped heatmaps (which are effectively ‘expert opinions’ about joint positions from the past and future) are then pooled into a single heatmap for each joint, from which the pose is estimated as the maximum. We next discuss the architecture of the ConvNets in detail. This is followed by a description of how optical flow is used to warp and pool the output from the Spatial ConvNet. 2.1. Spatial ConvNet The network is trained to regress the location of the human joint positions. However, instead of regressing the joint (x, y) positions directly [24, 37], we regress a heatmap of the joint positions, separately for each joint in an input image. This heatmap (the output of last convolutional layer, conv8) is a fixed-size i × j × k-dimensional cube (here 64 × 64 × 7 for k = 7 upper-body joints). At training time, the ground truth label are heatmaps synthesised for each joint separately by placing a Gaussian with fixed variance at the ground truth joint position (see Fig 2). We then Iteration 262,400 k joints Iteration 1,146,400 Figure 2. Regression target for learning the Spatial ConvNet. The learning target for the convolutional network is (for each of k joints) a heatmap with a synthesised Gaussian with a fixed variance centred at the ground truth joint position. The loss is l2 between this target and the output of the last convolutional layer. use an l2 loss, which penalises the squared pixel-wise differences between the predicted heatmap and the synthesised ground truth heatmap. We denote the training example as (X, y), where y stands for the coordinates of the k joints in the image X. Given training data N = {X, y} and the ConvNet regressor φ (output from conv8), the training objective becomes the task of estimating the network weights λ: X X kGi,j,k (yk ) − φi,j,k (X, λ)k2 (1) arg min λ (X,y)∈N i,j,k 1 2 2 2 1 −[(yk −i) +(yk −j) where Gi,j,k (yi ) = 2πσ 2e sian centred at joint yk with fixed σ. ]/2σ 2 is a Gaus- Discussion. As noted by [36], regressing coordinates directly is a highly non-linear and more difficult to learn mapping, which we also confirm here (Sect 5). The benefits of regressing a heatmap rather than (x, y) coordinates are twofold: first, one can understand failures and visualise the ‘thinking process’ of the network (see Figs 3 and 5); second, since by design, the output of the network can be multimodal, i.e. allowed to have confidence at multiple spatial locations, learning becomes easier: early on in training (as shown in Fig 3), multiple locations may fire for a given joint; the incorrect ones are then slowly suppressed as training proceeds. In contrast, if the output were only the wrist (x, y) coordinate, the net would only have a lower loss if it gets its prediction right (even if it was ‘growing confidence’ in the correct position). Architecture. The network architecture is shown in Fig 1, and sample activations for the layers are shown in Fig 5. To maximise the spatial resolution of the heatmap we make two important design choices: (i) minimal pooling is used (only two 2 × 2 max-pooling layers), and (ii) all strides are Figure 3. Multiple peaks are possible with the Spatial ConvNet. Early on in training (top), multiple locations may fire for a given joint. These are then suppressed as training proceeds (bottom). The arrows identify two modes for the wrist; one correct, one erroneous. As the training proceeds the erroneous one is diminished. unity (so that the resolution is not reduced). All layers are followed by ReLUs except conv9 (the pooling layer). In contrast to AlexNet [21], our network is fully convolutional (no fully-connected layers) with the fully-connected layers of [21] replaced by 1 × 1 convolutions. In contrast to both AlexNet and [35], our network is deeper, does not use local contrast normalisation (as we did not find this beneficial), and utilises less max-pooling. 2.2. Spatial fusion layers Vanilla heatmap pose nets do not learn spatial dependencies of joints, and thus often predict kinematically impossible poses (see examples in the extended arXiv version of this paper). To address this, we add what we term ‘spatial fusion layers’ to the network. These spatial fusion layers (normal convolutional layers) take as an input pre-heatmap activations (conv7), and learn dependencies between the human body parts locations represented by these activations. In detail, these layers take as an input a concatenation of conv7 and conv3 (a skip layer), and feed these through five more convolutional layers with ReLUs (see Fig 4). Large kernels are used to inflate the receptive field of the network. We attach a separate loss layer to the end of this network and backpropagate through the whole network. 2.3. Optical flow for pose estimation Given the heatmaps from the Spatial ConvNet from multiple frames, the heatmaps are reinforced with optical flow. This is done in three steps: (1) the confidences from nearby SpatialNet conv1 conv2 conv3 conv4 conv5 5x5x256 9x9x512 conv8 5x5x128 pool 2x2 5x5x128 conv6 conv7 5x5x128 pool 2x2 1x1x256 1x1x256 1x1x7 Loss 1 Spatial fusion layers conv1_f conv2_f conv3_f conv4_f conv5_f 7x7x64 13x13x64 13x13x128 1x1x256 1x1x7 Loss 2 Figure 4. Spatial fusion layers. The fusion layers learn to encode dependencies between human body parts locations, learning an implicit spatial model. then simply the positions of maximum confidence from the composite map. Below we discuss the first two steps. Single input frame conv1 conv5 conv2 conv6 conv3 conv7 conv4 conv8 (output) Figure 5. Sample activations for convolutional layers. Neuron activations are shown for three randomly selected channels for each convolutional layer (resized here to the same size), with the input (pre-segmented for visualisation purposes) shown above. Low down in the net, neurons are activated at edges in the image (e.g. conv1 and conv2); higher up, they start responding more clearly to body parts (conv6 onwards). The outputs in conv8 are shown for the right elbow, left shoulder and left elbow. frames are aligned to the current frame using dense optical flow; (2) these confidences are then pooled into a composite confidence map using an additional convolutional layer; and (3) the final upper body pose estimate for a frame is Step 1: Warping confidence maps with optical flow. For a given frame t, pixel-wise temporal tracks are computed from all neighbouring frames within n frames from ((t − n) to (t + n)) to frame t using dense optical flow [38]. These optical flow tracks are used to warp confidence values in neighbouring confidence maps to align them to frame t by effectively shifting confidences along the tracks [4]. Example tracks and the warping of wrist confidence values are shown in Fig 6. Step 2: Pooling the confidence maps. The output of Step 1 is a set of confidence maps that are warped to frame t. From these ‘expert opinions’ about the joint positions, the task is first to select a confidence for each pixel for each joint, and then to select one position for each joint. One solution would be to simply average the warped confidence maps. However, not all experts should be treated equally: intuitively, frames further away (thus with more space for optical flow errors) should be given lower weight. To this end we learn a parametric cross-channel pooling layer that takes as an input a set of warped heatmaps for a given joint, and as an output predicts a single ‘composite heatmap’. The input to this layer is a i × j × t heatmap volume, where t is the number of warped heatmaps (e.g. 31 for a neighbourhood of n = 15). As the pooling layer, we train a 1 × 1 kernel size convolutional layer for each joint. This is equivalent to cross-channel weighted sum-pooling, where we learn a single weight for each input channel (which correspond to the warped heatmaps). In total, we therefore learn t × k weights (for k joints). Right wrist (correct) Right wrist (wrong) (a) Frame t (b) Confidence at t (c) Warping (d) Pooled at t Figure 6. Warping neighbouring heatmaps for improving pose estimates. (a) RGB input at frame t. (b) Heatmap at frame t for right hand in the image. (c) Heatmaps from frames (t − n) and (t + n) warped to frame t using tracks from optical flow (green & blue lines). (d) Pooled confidence map with corrected modes. Train frames Test frames Train labels Test labels Train videos Val videos Test videos BBC 1.5M 1,000 [2] Manual 13 2 5 Ext. BBC 7M 1,000 [2, 3] Manual 85 2 5 ChaLearn 1M 3,200 Kinect Kinect 393 287 275 PiW 4.5K (FLIC) 830 Manual Manual 30 Table 1. Dataset overview, including train/val/test splits. 3. Implementation Details Training. The input frames are rescaled to height 256. A 248 × 248 sub-image (of the N × 256 input image) is randomly cropped, randomly horizontally flipped, randomly rotated between −40◦ and −40◦ , and resized to 256 × 256. Momentum is set to 0.95. The variance of the Gaussian is set to σ = 1.5 with an output heatmap size of 64 × 64. A temporal neighbourhood of n = 15 is input into the parametric pooling layer. The learning rate is set to 10−4 , and decreased to 10−5 at 80K iterations, to 10−6 after 100K iterations and stopped at 120K iterations. We use Caffe [19]. overlaid sign language interpreter. Each frame has been assigned pose estimates using the semi-automatic but reliable pose estimator of Buehler et al. [2] (used as training labels). 1,000 frames in the dataset have been manually annotated with upper-body pose (used as testing labels). Extended BBC Pose dataset. This dataset [24] contains 72 additional training videos which, combined with the original BBC TV dataset, yields in total 85 training videos. The frames of these new videos have been assigned poses using the automatic tracker of Charles et al. [3]. The output of this tracker is noisier than the semi-automatic tracker of Buehler et al., which results in partially noisy annotations. ChaLearn dataset. The ChaLearn 2013 Multi-modal gesture dataset [9] contains 23 hours of Kinect data of 27 people. The data includes RGB, depth, foreground segmentations and full body skeletons. In this dataset, both the training and testing labels are noisy (from Kinect). The large variation in clothing across videos poses a challenging task for pose estimation methods. Optical flow. Optical flow is computed using FastDeepFlow [38] with Middlebury parameters. Poses in the Wild (PiW) and FLIC datasets. The Poses in the Wild dataset [6] contains 30 sequences (total 830 frames) extracted from Hollywood movies. The frames are annotated with upper-body poses. It contains realistic poses in indoor and outdoor scenes, with background clutter, severe camera motion and occlusions. For training, we follow [6] and use all the images annotated with upper-body parts (about 4.5K) in the FLIC dataset [26]. 4. Datasets 5. Experiments Experiments in this work are conducted on four large video pose estimation datasets, two from signed TV broadcasts, one of Italian gestures, and the third of Hollywood movies. An overview of the datasets is given in Table 1. The first three datasets are available at http://www. robots.ox.ac.uk/˜vgg/data/pose. BBC Pose dataset. This dataset [3] consists of 20 videos (each 0.5h–1.5h in length) recorded from the BBC with an We first describe the evaluation protocol, then present comparisons to alternative network architectures, and finally give a comparison to state of the art. A demo video is online at http://youtu.be/pj2N5DqBOgQ. Training time. Training was performed on four NVIDIA GTX Titan GPUs using a modified version of the Caffe framework [19] with multi-GPU support. Training SpatialNet on FLIC took 3 days & SpatialNet Fusion 6 days. 5.1. Evaluation protocol and details Evaluation protocol. In all pose estimation experiments we compare the estimated joints against frames with manual Wrists Baseline method. As a baseline method we include a CoordinateNet (described in [23]). This is a network with similar architecture to [28], but trained for regressing the joint positions directly (instead of a heatmap) [24]. Computation time. Our method is real-time (50fps on 1 GPU without optical flow, 5fps with optical flow). 5.2. Component evaluation For these experiments the SpatialNet and baseline are trained and tested on the BBC Pose and Extended BBC Pose datasets. Fig 7 shows the results for wrists With the SpatialNet, we observe a significant boost in performance (an additional 6.6%, from 79.6% to 86.1% at d = 6) when training on the larger Extended BBC dataset compared to the BBC Pose dataset. As noted in Sect 4, this larger dataset is somewhat noisy. In contrast, the CoordinateNet is unable to make effective use of this additional noisy training data. We believe this is because its target (joint coordinates) does not allow for multi-modal output, which makes learning from noisy annotation challenging. We observe a further boost in performance from using optical flow to warp heatmaps from neighbouring frames (an improvement of 2.6%, from 86.1% to 88.7% at d = 6). Fig 9 shows the automatically learnt pooling weights. We see that for this dataset, as expected, the network learns to weigh frames temporally close to the current frame higher (because they contain less errors in optical flow). Fig 8 shows a comparison of different pooling types (for cross-channel pooling). We compare learning a parametric pooling function to sum-pooling and to max-pooling (maxout [14]) across channels. As expected, parametric pooling performs best, and improves as the neighbourhood n increases. In contrast, results with both sum-pooling and max-pooling deteriorate as the neighbourhood size is increased further, as they are not able to down-weigh predictions that are further away in time (and thus more prone to errors in optical flow). As expected, this effect is particularly noticeable for max-pooling. Failure modes. The main failure mode for the vanilla heatmap network (conv1-conv8) occurs when multiple modes are predicted and the wrong one is selected (and the 90 80 70 60 50 40 CoordinateNet CoordinateNet Extended SpatialNet Extended SpatialNet Flow Extended SpatialNet 30 20 10 0 0 2 4 6 8 10 12 Distance from GT [px] 14 16 18 20 Figure 7. Comparison of the performance of our nets for wrists on BBC Pose. Plots show accuracy as the allowed distance from manual ground truth is increased. CoordinateNet is the network in [23]; SpatialNet is the heatmap network; and SpatialNet Flow is the heatmap network with the parametric pooling layer. ‘Extended’ indicates that the network is trained on Extended BBC Pose instead of BBC Pose. We observe a significant gain for the SpatialNet from using the additional training data in the Extended BBC dataset (automatically labelled – see Sect 4) training data, and a further boost from using optical flow information (and selecting the warping weights with the parametric pooling layer). Wrists 84 82 Accuracy [%] at d=5px Experimental details. All frames of the training videos are used for training (with each frame randomly augmented as detailed above). The frames are randomly shuffled prior to training to present maximally varying input data to the network. The hyperparameters (early stopping, variance σ etc.) are estimated using the validation set. 100 Accuracy [%] ground truth (except ChaLearn, where we compare against output from Kinect). We present results as graphs that plot accuracy vs distance from ground truth in pixels, where a joint is deemed correctly located if it is within a set distance of d pixels from a marked joint centre in ground truth. 80 78 76 74 Sumpool Maxout Parametric pooling 72 70 0 2 4 6 8 10 Neighbourhood size (n) 12 14 Figure 8. Comparison of pooling types. Results are shown for wrists in BBC Pose at threshold d = 5px. Parametric pooling (learnt cross-channel pooling weights) performs best. resulting poses are often kinematically impossible for a human to perform). Examples of these failures are included in the extended arXiv version of this paper. The spatial fusion layers resolve these failures. 5.3. Comparison to state of the art Training. We investigated a number of strategies for training on these datasets including training from scratch (using only the training data provided with the dataset), or training on one (i.e. BBC Pose) and fine-tuning on the others. We found that provided the first and last layers of the Spatial Net are initialized from (any) trained heatmap network, the rest can be trained either from scratch or finetuned with similar performance. We hypothesise this is because the datasets are very different – BBC Pose contains long-sleeved persons, ChaLearn short-sleeved persons and Wrists 100 90 0.25 70 0.15 60 Accuracy [%] pooling weight 80 0.2 0.1 50 40 0.05 Charles et al. (2013) Charles et al. (2014) Yang & Ramanan (2013) SpatialNet Flow SpatialNet Fusion Flow SpatialNet 30 0 −15 20 −10 −5 0 frame number 5 10 10 15 Figure 9. Learnt pooling weights for BBC Pose with n = 15. Weights shown for the right wrist. The centre frame receives highest weight. The jitter in weights is due to errors in optical flow computation (caused by the moving background in the video) – the errors become larger further away from the central frame (hence low or even negative weights far away). 0 0 2 4 6 8 10 12 Distance from GT [px] 14 16 18 20 Figure 11. Comparison to the state of the art on ChaLearn. Our method outperforms state of the art by a large margin (an addition of 19% at d = 4). See arXiv version for elbows & shoulders. Wrists 80 Chalearn. Fig 11 shows a comparison to the state of the art on ChaLearn. We again outperform the state of the art even without optical flow (an improvement of 3.5% at d = 6), and observe a further boost by using optical flow (beating state of the art by an addition of 5.5% at d = 6), and a significant further improvement from using a deeper network (an additional 13% at d = 6). 60 Accuracy [%] BBC Pose. Fig 10 shows a comparison to the state of the art on the BBC Pose dataset. We compare against all previous reported results on the dataset. These include Buehler et al. [2], whose pose estimator is based on a pictorial structure model; Charles et al. (2013) [3] who uses a Random Forest; Charles et al. (2014) [4] who predict joints sequentially with a Random Forest; Pfister et al. (2014) [24] who use a deep network similar to our CoordinateNet (with multiple input frames); and the deformable part-based model of Yang & Ramanan (2013) [39]. We outperform all previous work by a large margin, with a particularly noticeable gap for wrists (an addition of 10% compared to the best competing method at d = 6). 70 50 40 30 Cherian et al. (2014) Yang & Ramanan (2013) SpatialNet Fusion Flow SpatialNet Fusion SpatialNet 20 10 0 0 2 4 6 8 10 12 Distance from GT [px] 14 16 FLIC. Fig 13 shows a comparison to the state of the art on FLIC. We outperform all pose estimation methods that 20 Average PCK for wrist & elbow 100 90 80 70 60 50 Toshev et al. Jain et al. MODEC Yang et al. Sapp et al. Tompson et al. * Chen et al. * Ours 40 30 Poses in the Wild. Figs 12 & 14 show a comparison to the state of the art on Poses in the Wild. We replicate the results of the previous state of the art method using code provided by the authors [6]. We outperform the state of the art on this dataset by a large margin (an addition of 30% for wrists and 24% for elbows at d = 8). Using optical flow yields a significant 10% improvement for wrists and 13% for elbows at d = 8. Fig 15 shows example predictions. 18 Figure 12. Comparison to state of the art on Poses in the Wild. Our method outperforms state of the art by a large margin (an addition of 30% at d = 8, with 10% from flow). Accuracy [%] Poses in the Wild contains non-frontal poses with unusual viewing angles. For all the results reported here we train BBC Pose from scratch, initialize the first and last layer from this, and fine-tune on training data of other datasets. 20 10 0 0 0.025 0.05 0.075 0.1 0.125 Normalised distance from GT 0.15 0.175 0.2 Figure 13. Comparison to state of the art on FLIC. Solid lines represent deep models; methods with a square () are without a graphical model; methods with an asterisk (*) are with a graphical model. Our method outperforms competing methods without a graphical model by a large margin in the high precision area (an addition of 20% at d = 0.05). Wrists Elbows Shoulders Average 100 90 90 90 90 90 80 80 80 80 80 70 70 70 70 70 60 50 40 30 60 50 40 Buehler et al. (2011) Charles et al. (2013) Charles et al. (2014) Pfister et al. (2014) Yang & Ramanan (2013) Ours 20 10 0 0 60 50 40 Accuracy [%] 100 Accuracy [%] 100 Accuracy [%] 100 Accuracy [%] Accuracy [%] Head 100 60 50 40 60 50 40 30 30 30 30 20 20 20 20 10 10 10 10 0 0 5 10 15 20 Distance from GT [px] 5 10 15 20 Distance from GT [px] 0 0 5 10 15 20 Distance from GT [px] 0 0 5 10 15 20 Distance from GT [px] 0 0 5 10 15 20 Distance from GT [px] Figure 10. Comparison to the state of the art on BBC Pose. Plots show accuracy per joint type (average over left and right body parts) as the allowed distance from manual ground truth is increased. We outperform all previous work by a large margin; notice particularly the performance for wrists, where we outperform the best competing method with an addition of 10% at d = 6. Our method uses SpatialNet Flow Extended. Pfister et al. (2014) uses Extended BBC Pose; Buehler et al., Charles et al. and Yang & Ramanan use BBC Pose. Shoulders 100 90 90 80 80 70 70 60 60 Accuracy [%] Accuracy [%] Elbows 100 50 40 30 40 30 Cherian et al. (2014) Yang & Ramanan (2013) SpatialNet Fusion Flow SpatialNet Fusion SpatialNet 20 10 0 0 50 5 10 15 20 25 Distance from GT [px] 30 20 10 0 0 5 10 15 20 25 Distance from GT [px] 30 Figure 14. Poses in the Wild: elbows & shoulders. don’t use a graphical model, and match or even slightly outperform graphical model-based methods [5, 35] in the very high precision region (< 0.05 from GT). The increase in accuracy at d = 0.05 is 20% compared to methods not using a graphical model, and 12% compared to [5] who use a graphical model. Tompson et al. is [35]; Jain et al. is [17]. Predictions are provided by the authors of [5, 35] and evaluation code by the authors of [35]. 6. Conclusion We have presented a new architecture for pose estimation in videos that is able to utilizes appearances across multiple frames. The proposed ConvNet is a simple, direct method for regressing heatmaps, and its performance is improved by combining it with optical flow and spatial fusion layers. We have also shown that our method outperforms the state of the art on three large video pose estimation datasets. Figure 15. Example predictions on Poses in the Wild. Further improvements may be obtained by using additional inputs for the spatial ConvNet, for example multiple RGB frames [24] or optical flow [18] – although prior work has shown little benefit from this so far. The benefits of aligning pose estimates from multiple frames using optical flow, as presented here, are complementary to architectures that explicitly add spatial MRF and refinement layers [35, 36]. Finally, we have demonstrated the architecture for human pose estimation, but a similar optical flow-mediated combination of information could be used for other tasks in video, including classification and segmentation. Acknowledgements: Financial support was provided by Osk. Huttunen Foundation and EPSRC grant EP/I012001/1. References [1] L. Bourdev and J. Malik. Poselets: Body part detectors trained using 3d human pose annotations. In Proc. CVPR, 2009. 1 [2] P. Buehler, M. Everingham, D. P. Huttenlocher, and A. Zisserman. Upper body detection and tracking in extended signing sequences. IJCV, 2011. 1, 5, 7 [3] J. Charles, T. Pfister, M. Everingham, and A. Zisserman. Automatic and efficient human pose estimation for sign language videos. IJCV, 2013. 1, 5, 7 [4] J. Charles, T. Pfister, D. Magee, D. Hogg, and A. Zisserman. Upper body pose estimation with temporal sequential forests. Proc. BMVC, 2014. 1, 2, 4, 7 [5] X. Chen and A. Yuille. Articulated pose estimation with image-dependent preference on pairwise relations. In Proc. NIPS, 2014. 1, 8 [6] A. Cherian, J. Mairal, K. Alahari, and C. Schmid. Mixing body-part sequences for human pose estimation. In Proc. CVPR, 2014. 5, 7 [7] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. Proc. ICML, 2014. 1 [8] M. Eichner, M. Marin-Jimenez, A. Zisserman, and V. Ferrari. 2d articulated human pose estimation and retrieval in (almost) unconstrained still images. IJCV, 2012. 1 [9] S. Escalera, J. Gonzalez, X. Baro, M. Reyes, O. Lopes, I. Guyon, V. Athistos, and H. Escalante. Multi-modal gesture recognition challenge 2013: Dataset and results. In Proc. ICMI, 2013. 5 [10] P. Felzenszwalb and D. Huttenlocher. Pictorial structures for object recognition. IJCV, 2005. 1 [11] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. CVPR, 2014. 1 [12] R. Girshick, J. Shotton, P. Kohli, A. Criminisi, and A. Fitzgibbon. Efficient regression of general-activity human poses from depth images. In Proc. ICCV, 2011. 1 [13] G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik. Using k-poselets for detecting people and localizing their keypoints. In Proc. CVPR, 2014. 1 [14] I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. J. Machine Learning Research, 2013. 6 [15] I. J. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud, and V. Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. In Proc. ICLR, 2014. 1 [16] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. Proc. NIPS Workshops, 2014. 1 [17] A. Jain, J. Tompson, M. Andriluka, G. Taylor, and C. Bregler. Learning human pose estimation features with convolutional networks. Proc. ICLR, 2014. 1, 8 [18] A. Jain, J. Tompson, Y. LeCun, and C. Bregler. MoDeep: A deep learning framework using motion features for human pose estimation. Proc. ACCV, 2014. 2, 8 [19] Y. Jia. Caffe: An open source convolutional architecture for fast feature embedding. http://caffe. berkeleyvision.org/, 2013. 5 [20] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proc. CVPR, 2014. 1 [21] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Proc. NIPS, 2012. 1, 3 [22] M. Osadchy, Y. LeCun, and M. Miller. Synergistic face detection and pose estimation with energy-based models. JMLR, 2007. 1 [23] T. Pfister. Advancing Human Pose and Gesture Recognition. PhD thesis, University of Oxford, 2015. 6 [24] T. Pfister, K. Simonyan, J. Charles, and A. Zisserman. Deep convolutional neural networks for efficient pose estimation in gesture videos. Proc. ACCV, 2014. 1, 2, 5, 6, 7, 8 [25] S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. CVPR Workshops, 2014. 1 [26] B. Sapp and B. Taskar. Modec: Multimodal decomposable models for human pose estimation. In Proc. CVPR, 2013. 5 [27] B. Sapp, D. Weiss, and B. Taskar. Parsing human motion with stretchable models. In Proc. CVPR, 2011. 1 [28] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. Proc. ICLR, 2014. 1, 6 [29] J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Criminisi, A. Kipman, and A. Blake. Efficient human pose estimation from single depth images. PAMI, 2013. 1 [30] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. Proc. NIPS, 2014. 1, 2 [31] M. Sun, P. Kohli, and J. Shotton. Conditional regression forests for human pose estimation. In Proc. CVPR, 2012. 1 [32] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proc. CVPR, 2014. 1 [33] G. Taylor, R. Fergus, G. Williams, I. Spiro, and C. Bregler. Pose-sensitive embedding by nonlinear nca regression. In Proc. NIPS, 2010. 1 [34] J. Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon. The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In Proc. CVPR, 2012. 1 [35] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. Efficient object localization using convolutional networks. Proc. CVPR, 2015. 1, 3, 8 [36] J. Tompson, A. Jain, Y. LeCun, and C. Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. Proc. NIPS, 2014. 1, 3, 8 [37] A. Toshev and C. Szegedy. DeepPose: Human pose estimation via deep neural networks. CVPR, 2014. 1, 2 [38] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. Deepflow: Large displacement optical flow with deep matching. In Proc. ICCV, 2013. 4, 5 [39] Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. PAMI, 2013. 1, 7 [40] M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. Proc. ECCV, 2014. 1 Understanding deep features with computer-generated imagery Mathieu Aubry École des Ponts ParisTech UC Berkeley Bryan C. Russell Adobe Research mathieu.aubry@imagine.enpc.fr brussell@adobe.com Abstract not possible, e.g. due to lack of labeled data. Prior work has focused on a part-based analysis of the learned convolutional filters. Examples include associating filters with input image patches having maximal response [12], deconvolution starting from a given filter response [38], or by masking the input to recover the receptive field of a given filter [39] to generate “simplified images” [6, 31]. Such visualizations typically reveal the parts of an object [38] (e.g. “eye” of a cat) or scene [39] (e.g. “toilet” in bathroom). While these visualizations reveal the nature of learned filters, they largely ignore the question of the dependence of the CNN representation on continuous factors that may influence the depicted scene, such as 3D viewpoint, scene lighting configuration, and object style. In this paper, we study systematically how different scene factors that arise in natural images are represented in a trained CNN. Example factors may include those intrinsic to an object or scene, such as category, style, and color, and extrinsic ones, such as 3D viewpoint and scene lighting configuration. Studying the variations associated with such factors is a nontrivial task as it requires (i) input data where the factors can be independently controlled and (ii) a procedure for detecting, visualizing, and quantifying each factor in a trained CNN. To overcome the challenges associated with obtaining input data, we leverage computer-generated (CG) imagery to study trained CNNs. CG images offer several benefits. First, there are stores of 3D content online (e.g. Trimble 3D Warehouse), with ongoing efforts to curate and organize the data for research purposes (e.g. ModelNet [35]). Such data spans many different object categories and styles. Moreover, in generating CG images we have control over all rendering parameters, which allows us to systematically and densely sample images for any given factor. A database of natural images captured in controlled conditions and spanning different factors of variations, e.g. the NORB [22], ETH-80 [23] and RGB-D object [20] datasets, where different objects are rotated on a turntable and lighting is varied during image capture, are difficult and costly to collect. Moreover they do not offer the same variety of object styles present in 3D model collections, nor the flexibility given by rendering. We introduce an approach for analyzing the variation of features generated by convolutional neural networks (CNNs) with respect to scene factors that occur in natural images. Such factors may include object style, 3D viewpoint, color, and scene lighting configuration. Our approach analyzes CNN feature responses corresponding to different scene factors by controlling for them via rendering using a large database of 3D CAD models. The rendered images are presented to a trained CNN and responses for different layers are studied with respect to the input scene factors. We perform a decomposition of the responses based on knowledge of the input scene factors and analyze the resulting components. In particular, we quantify their relative importance in the CNN responses and visualize them using principal component analysis. We show qualitative and quantitative results of our study on three CNNs trained on large image datasets: AlexNet [18], Places [40], and Oxford VGG [8]. We observe important differences across the networks and CNN layers for different scene factors and object categories. Finally, we demonstrate that our analysis based on computergenerated imagery translates to the network representation of natural images. 1. Introduction The success of convolutional neural networks (CNNs) [18, 21] raises fundamental questions on how their learned representations encode variations in visual data. For example, how are different layers in a deep network influenced by different scene factors, the task for which the network was trained for, or the choice in network architecture? These questions are important as CNNs with different architectures and trained/fine tuned for different tasks have shown to perform differently [17, 40] or have different feature response characteristics [39]. An analysis of the features may help with understanding the tradeoffs across different trained networks and may inform the design of new architectures. It may also help the choice of CNN features for tasks where training or fine tuning a network is 1 Given a set of rendered images generated by varying one or more factors, we analyze the responses of a layer for a trained CNN (e.g. “pool5” of AlexNet [18]). We perform a decomposition of the responses based on knowledge of the input scene factors, which allows us to quantify the relative importance of each factor in the representation. Moreover, we visualize the responses via principal component analysis (PCA). Contributions. Our technical contribution is an analysis of contemporary CNNs trained on large-scale image datasets via computer generated images, particularly rendered from 3D models. From our study we observe: • Features computed from image collections that vary along two different factors, such as style and viewpoint, can often be approximated by a linear combination of features corresponding to the factors, especially in the higher layers of a CNN. • Sensitivity to viewpoint decreases progressively in the last layers of the CNNs. Moreover, the VGG fc7 layer appears to be less sensitive to viewpoint than AlexNet and Places. • Relative to object style, color is more important for the Places CNN than for AlexNet and VGG. This difference is more pronounced for the background color than for foreground. • The analysis we perform on deep features extracted from rendered views of 3D models is related to understanding their representation in natural images. 1.1. Related work In addition to the prior work to visualize learned CNN filters [5, 11, 12, 30, 38, 39], there has been work to visualize hand-designed [34] and deep [24] features. Also related are recent work to understand the quantitative tradeoffs across different CNN layers for networks trained on large image databases [1, 37] and designing CNN layers manually [7]. Our use of a large CAD model dataset can be seen in the context of leveraging such data for computer vision tasks, e.g. object detection [2]. Contemporary approaches have used synthetic data with CNNs to render images for particular scene factors, e.g. style, pose, lighting [10, 19]. Our PCA feature analysis is related to prior work on studying visual embeddings. The classic Eigenfaces paper [33] performed PCA on faces. Later work studied nonlinear embeddings, such as LLE [27] and IsoMap [32]. Most related to us is the study of nonlinear CNN feature embeddings with the NORB dataset [14]. In contrast we study large-scale, contemporary CNNs trained on large image datasets. Finally, our multiple factor study is related to intrinsic image decomposition [4]. We note a contemporary approach for separating style and content via autoencoders [9]. 1.2. Overview Our deep feature analysis begins by rendering a set of stimuli images by varying one or more scene factors. We present the stimuli images to a trained CNN as input and record the feature responses for a desired layer. Given the feature responses for the stimuli images we analyze the principal modes of variation in the feature space via PCA (section 2.1). When more than one factor is present we linearly decompose the feature space with respect to the factors and perform PCA on the feature decomposition (section 2.2). We give details of our experimental setup in section 3 and show qualitative and quantitative results over a variety of synthetic and natural images in section 4. 2. Approach for deep feature analysis In this section we describe our approach for analyzing the image representation learned by a CNN. We seek to study how the higher levels of the CNN encodes the diversity present in a set of images. The minimal input for our analysis is a set of related images. We first describe our approach for analyzing jointly their features. What we can learn with such an approach is however limited since it cannot identify the origin of the variations of the input images. The factors of variation can be, e.g., variations in the style of an object, changes in its position, scaling, 3D rotation, lighting, or color. For this reason, we then focus on the case when the images are computer generated and we have full control of the different factors. In this case, we seek to separate the influence of the different factors on the representation, analyze them separately, and compare their relative importance. 2.1. Image collection analysis We seek to characterize how a CNN encodes a collection of related images, Ω, e.g. images depicting a “car” or a black rectangle on white background. We sample images rθ ∈ Ω indexed by θ ∈ Θ. In the case of natural images Θ is an integer index set over the collection Ω. In the case of computer-generated images Θ is a set of parameters corresponding to a scene factor we wish to study (e.g. azimuth and elevation angles for 3D viewpoint, 3D model instances for object style, or position of the object in the image for 2D translation). Given a trained CNN, let F̃ L (rθ ) be a column vector of CNN responses for layer L (e.g. “pool5”, “fc6”, or “fc7” in AlexNet [18]) to the input image rθ . The CNN responses F̃ L are high-dimensional feature vectors that represent the image information. However, since Ω contains related images, we expect the features to be close to each other and their intrinsic dimension to be smaller than the actual feature dimension. For this reason, we use principal component analysis (PCA) [25] to identify a set of orthonormal basis vectors that capture the principal modes of variation of the features. Given centered features F L (θ) = F̃ L (rθ ) − P 1 L t∈Θ F̃ (rt ), where |X| is the number of ele|Θ| ments in set X, we compute the eigenvectors associatedP with the largest eigenvalues of the covariance matrix 1 L L T θ∈Θ F (θ)(F (θ)) . The projection of the features |Θ| onto the subspace defined by the D components with maximal eigenvalues corresponds to an optimal D-dimensional linear approximation of the features. We evaluate the intrinsic dimensionality of the features by computing the number of dimensions necessary to explain 95% of the variance. Moreover, we can visualize the embedding of the images by projecting onto the components with high variance. 2.2. Multiple factor analysis In the case when we have control of the variation parameters, we can go further and attempt to decompose the features as a linear combination of uncorrelated components associated to the different factors of variation. Features decomposing linearly into different factors would be powerful and allow to perform image transformations directly in feature space and, e.g., to compare easily images taken under different viewpoints. Let Θ1 , . . . , ΘN be sets of parameters for N factors of variation we want to study. We consider an image of the scene with parameters θ = (θ1 , . . . , θN ), where θ ∈ Θ = Θ1 ×· · ·×ΘN . We assume the θk are sampled independently. We define marginal features FkL (θk ) for scene factor k by marginalizing over the parameters for all factors except k: FkL (t) = = E(F L (θ)|θk = t) |Θk | X F L (θ) |Θ| (1) (2) θ∈Θ|θk =t Similar to section 2.1, we can study via PCA the principal modes of variation over the marginal features FkL for each factor k. Finally, we define a residual feature ∆L (θ), which is the difference of the centered CNN features F L (θ) and the sum of all the marginal features FkL (θk ). This results in the following decomposition: L F (θ) = N X FkL (θk ) + ∆L (θ) (3) k=1 Using computer-generated images, we can easily compute this decomposition by rendering the images with all the rendering parameters corresponding to the sum in equation (2). Direct computation shows that all the terms in decomposition (3) have zero mean and are uncorrelated. This implies: var(F L ) = N X k=1 var(FkL ) + var(∆L ) (4) We can thus decompose the variance of the features F L as the sum of the variances associated to the different factors and a residual. When analyzing the decomposed features we report the relative variance RkL = var(FkL )/var(F L ). L We also report the relative variance of the residual R∆ = L L var(∆ )/var(F ). A factor’s relative variance provides an indication of how much the factor is represented in the CNN layer compared to the others. A high value indicates the factor is dominant in the layer and conversely a low value indicates the factor is largely negligible compared to the others. Moreover, a low value of the residual relative variance indicates the are largely separated in the layer. Note Pfactors N L L that R∆ + k=1 RkL = 1 and the values of RkL and R∆ do not depend on the relative sampling of the different factors. 3. Experimental setup In this section we describe details of our experimental setup. In particular we describe the details of the CNN features we extract, our rendering pipeline, and the set of factors we seek to study. 3.1. CNN features We study three trained CNN models: AlexNet [18], winner of the 2012 ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) [28], Places [40], which has the same architecture as AlexNet but trained on a large image database depicting scenes, and Oxford VGG [8] CNN-S network. In particular, we study the features of the higher layers “pool5”, “fc6”, and “fc7” of these networks. Note that the Oxford VGG architecture is different and the dimension of its “pool5” layer is two times larger than AlexNet and Places. We use the publicly-available CNN implementation of Caffe [16] to extract features for the different layers and pre-trained models for AlexNet, Places, and Oxford VGG from their model zoo. 3.2. Computer-generated imagery We present two types of image stimuli as input: (i) 2D abstract stimuli consisting of constant color images or rectangular patches on constant background, which are described in detail in section 4.1, and (ii) rendered views from 3D models. For the latter we seek to render different object categories spanning many different styles from a variety of viewpoints and under different illumination conditions. We used as input CAD 3D models from the ModelNet database [35], which contains many models having different styles for a variety of object classes. We downloaded the CAD models in Collada file format for the following object classes: chair (1261 models), car (485 models), sofa (701 models), toilet (191 models) and bed (258 models). We adapted the publicly-available OpenGL renderer from [3], which renders a textured CAD model with matte surfaces under fixed lighting configuration and allows the viewpoint to be specified by a 3×4 camera (a) AlexNet dims. 1, 2 (b) Places dims. 1, 2 (c) VGG dims. 1, 2 (d) VGG dims. 2, 3 Figure 1: PCA embeddings of constant-color images for the fc7 layer of different CNNs. The AlexNet and Places embeddings are similar to a hue color wheel, with more variation visible for the green and blue channels. Table 1: Relative variance of the aspect ratio, 2D position, and residual feature for our synthetic rectangle experiment with AlexNet. Notice that the relative variance of the aspect ratio increases with the higher layers while 2D position decreases, which indicates that the features focus more on the shape and less on the 2D location in the image. AlexNet, pool5 AlexNet, fc6 AlexNet, fc7 2D position 49.8 % 45.1 % 33.9 % Aspect ratio 9.5 % 22.3 % 37.0 % ∆L 40.8 % 32.6 % 29.1 % 4. Results matrix [15]. We render the models under different lighting conditions and with different uniform colors. We show results on two categories that have received the most attention in 3D-based image analysis: cars [13, 26], and chairs [2, 10]. Detailed results for three other categories are presented in the supplementary material and our quantitative results are averages over the 5 categories. 3.3. 3D scene factors We study two types of factors affecting the appearance of a scene: (i) intrinsic factors – object category, style, and color, and (ii) extrinsic factors – 2D position, 2D scale, 3D viewpoint, and scene lighting configuration. The object category and style factors are specified by the CAD models from ModelNet. For object color we study grayscale matte surfaces (specified by Lambertian surface model), and constant-colored matte surfaces with the color uniformly sampled on a grid in RGB colorspace. For the 2D extrinsic factors, we uniformly sample 2D positions along a grid in the image plane and vary the 2D scale linearly. For 3D viewpoint we manually aligned all the 3D models to a canonical coordinate frame, i.e. all models are consistently oriented with respect to gravity and face the same direction, orbit the object at constant 3D distance and uniformly sample the azimuth and elevation angles with respect to the object’s coordinate frame. Finally, we vary the scene lighting such that the source light varies from left to right and front to back on a uniform grid. For the quantitative experiments we sampled 36 azimuth angles (keeping the elevation fixed at 10 degrees) for rotation, 36 positions for translation, 40 scales, 36 light positions, and 125 colors. For the visualizations we sampled 120 azimuth angles, 121 light positions, and 400 positions to make the embeddings easier to interpret. We checked that the different sampling did not change our quantitative results, which was expected since our method is not sensitive to the relative number of samples for each factor. In this section we highlight a few results from our experiments on CNN feature analysis. The supplementary material reports our detailed quantitative results for all object categories and provides a visualization tool to interactively select and compare embeddings for the first ten PCA components. We first report results for manually-designed 2D stimuli in section 4.1 and then for rendered views of 3D models from several object categories in section 4.2. We finally show in section 4.3 that our results obtained with computer-generated images are related to natural images. 4.1. 2D abstract stimuli In this section we apply the deep feature analysis of section 2 on manually-designed 2D abstract stimuli presented to a trained CNN. We first perform a PCA analysis on a single factor, color. Next, we perform two-factor quantitative analyses on the aspect ratio/2D position of a rectangle and on the foreground/background colors of a centered square. Uniform color. We perform PCA on a set of images with constant color, as described in section 2.1. We sampled 1331 colors uniformly on a grid in RGB color space. The resulting embedding for the fc7 layers of the three CNNs are shown in figure 1. The resulting embeddings for the AlexNet and Places CNNs are surprisingly similar to a hue color wheel, with more variation visible for blues and greens and less for reds and violets. VGG has a different behavior, with the first dimension similar to saturation. The embedding corresponding to the second and third dimensions is similar to that of the other CNNs. The number of PCA dimensions necessary to explain 95% of the variance is approximately 20 for all three networks and all three layers, which is higher than the three dimensions of the input data. Position and aspect ratio. We used the methodology of section 2.2 to study the features representing a small black rectangle located at different positions with different aspect ratios on a white background. The rectangle area was kept (a) Chair, pool5 (e) Car, pool5 (b) Chair, pool5, style (f) Car, pool5, style (c) Chair, pool5, rotation (g) Car, pool5, rotation (d) Chair, fc6, rotation (h) Car, fc6, rotation Figure 2: Best viewed in the electronic version. PCA embeddings (dims. 1,2) of AlexNet features for “chairs” (first row) and “cars” (second row). Column 1 – Direct embedding of the rendered images without viewpoint-style separation. Columns 2,3 – Embeddings associated with style (for all rotations) and rotation (for all styles). Column 4 – Rotation embedding for fc6, which is qualitatively different than pool5. Colors correspond to orientation and can be interpreted via the example images in columns 3,4. Similar results for other categories and PCA dimensions are available in the supplementary material. constant at 0.262 of the image area. We consider the position and aspect ratio as two factors of variation and sample 36 positions on a grid and 12 aspect ratios on a log scale. The variance explained by each factor for the different layers of AlexNet is presented in table 1. For all three networks the relative variance associated to the position decreases, which quantitatively supports the idea that the higher layers have more translation invariance. In contrast the relative variance associated to the aspect ratio increases for the higher CNN layers. For AlexNet, less than 10% of the relative variance for pool5 is explained by the aspect ratio alone, while it explains 37% of the relative variance for fc7. Also, the relative variance associated with the residual decreases for the higher CNN layers, which indicates the two factors are more easily linearly separated in the higher layers. Center and surrounding color. Similar to the experiments presented in the previous paragraph, we considered a square of one color on a background of a different color. We chose the size of the central square to be half the image size. Quantitative results for the fc7 features of the different networks are presented in table 2. Results for the other lay- ers are in the supplementary material. A first observation is that the features do not separate as well the foreground and background colors in the representation as the aspect ratio and position in the previous experiment. We also observe that for all the networks the variance associated to the background color is higher than the variance associated to the foreground. The difference is more striking for the Places fc7 layer (3.8x versus 2x for AlexNet fc7 and 1.8x for VGG fc7). Future work could determine if the background color of an image is especially important for scene classification, while the foreground color is less important. Remarks. As the CNNs were not trained on the 2D artificial stimuli presented in this section, we find it somewhat surprising that the embeddings resulting from the above feature analysis is meaningful. From our experiments we saw that the CNNs learn a rich representation of colors, identifying in particular variations similar to hue and saturation. Moreover, the last layers of the network better encode translation invariance, focusing on shape. These results will be confirmed and generalized on more realistic stimuli in the next sections. Table 2: Relative variance and intrinsic dimensionality of a foreground square of one color on a background color. Each cell: top – rel. variance; bottom – intrinsic dim. Places, fc7 AlexNet, fc7 VGG, fc7 Foreground 13.4% 13 19.2% 14 20.2% 11 Background 51.1% 14 39.9% 16 36.9% 15 ∆L 35.5% 216 40.8% 315 42.9% 216 (a) Car, pool5 (b) Chair, pool5 (c) Car, fc6 (d) Chair, fc6 (e) Car, fc7 (f) Chair, fc7 4.2. Object categories In this section we want to explore the embedding generated by the networks for image sets and factors related to the tasks for which they are trained, namely object category classification in the case of AlexNet and VGG. We also compare against the CNN trained on Places. We thus select an object category and, using rendered views of 3D models, we analyze how the CNN features are influenced by the style of the specific instances as well as different transformations and rendering parameters. The parameter sampling for each experiment is described in section 3.3. Model–orientation separation. The first variation we study jointly with style is the rotation of the 3D model. The first column of figure 2 visualizes the PCA embedding of the resulting pool5 features. This embedding is hard to interpret because it mixes information about viewpoint (important for cars) and instance style (important for chairs). To separate this information, we perform the decomposition presented in section 2. The decomposition provides us with embedding spaces for style and viewpoint and associates to each model and viewpoint its own descriptor. We visualize the embeddings in figure 2; the second column corresponds to style and the third to viewpoint. Note that the different geometries of the two categories lead to different embeddings of rotation in pool5. While a left-facing car typically looks similar to a right-facing car and is close in the feature space (figure 2g), a right-facing chair is usually different from left-facing chairs and is far in the embedding (figure 2c). The last column shows the viewpoint embedding for fc6. The comparison of the last two columns indicates that much viewpoint information is lost between pool5 and fc6 and that fc6 is largely left-right flip invariant. A potential interesting future direction could be to interpret the viewpoint embeddings relative to classic work on mental rotation [29]. Translation, scale, lighting, color. We repeated the same experiment for the following factors: 2D translation, scale, light direction, background color, and object color. For Figure 3: PCA embeddings for 2D position on AlexNet. simplicity and computational efficiency, we considered in all experiments a frontal view of all the instances of the objects. The framework allows the same analysis using the object orientation as an additional factor. The embeddings associated with AlexNet features for translation of cars and chairs are shown in figure 3. Note that similar to rotations, the embedding corresponding to cars and chairs are different, and that the first two components of the fc6 features indicate a left-right flip-invariant representation. The embeddings for the pool5 layer of the car category for the other factors are shown figure 4. Quantitative analysis: viewpoint. We analyze the relative variance explained by the 3D rotation, translation, and scale experiments. While the variance was different for each factor and category, the variation across the layers and networks was consistent in all cases. For this reason we report in table 3 an average of the variance across all five categories and all three factors. We refer the reader to the supplementary material for detailed results. The analysis of table 3 reveals several observations. First, the proportion of the variance of deeper layers corresponding to viewpoint information is less important, while the proportion corresponding to style is more important. This corresponds to the Table 3: Relative variance and intrinsic dimensionality averaged over experiments for different object categories and viewpoints (3D orientation, translation, and scale). Each cell: top – rel. variance; bottom – intrinsic dim. We do not report the intrinsic dim. of ∆L since it is typically larger than 1K across the experiments and expensive to compute. (a) Lighting Places (b) Scale Viewpoint AlexNet VGG Places Style (c) Object color AlexNet (d) Background color VGG Figure 4: PCA embeddings for different factors using AlexNet pool5 features on “car” images. Colors in (a) correspond to location of the light source (green – center). intuition that higher layers are more invariant to viewpoint. We also note that the residual feature ∆L is less important in higher layers, indicating style and viewpoint are more easily separable in those layers. These observations are consistent with our results of section 4.1. Second, the part of the variance associated with style is more important in the fc7 layer for VGG than in AlexNet and Places. Also, the part associated with the viewpoint and residual is smaller. Note that this does not hold in pool5, where the residual is important for the VGG network. This effect may be related to the difference in the real and intrinsic dimension of the features. The intrinsic dimension of the style component of VGG pool5 features is larger and decreases from pool5 to fc7. On the contrary, the intrinsic dimensionality of AlexNet has smaller variation across layers. Finally, we note that the intrinsic dimensionality of the fc7 style feature of Places is smaller than the other networks. This may indicate that it is less rich, and may be related to the fact that identifying the style of an object is less crucial for scene classification. We believe it would be an interesting direction for future work to study how the improved performance of VGG for object classification is related to the observed reduced sensitivity to viewpoint. Quantitative analysis: color. We report in table 4 the average across categories of our quantitative study for object and background color. The results are different from those of viewpoint. First, we observe that a larger part of the variance of the features of the Places network is explained by the ∆L Places AlexNet VGG pool5 26.8 % 8.5 26.4 % 8.3 21.2 % 10.0 26.8 % 136.3 28.2 % 121.1 26.4 % 181.9 46.8 % 45.0 % 52.4 % fc6 21.4 % 7.0 19.4 % 7.2 16.4 % 7.7 39.1 % 105.5 40.3 % 125.5 44.3 % 136.3 39.5 % 40.3 % 39.3 % fc7 17.8 % 5.9 15.6 % 6.0 12.3 % 6.2 49.4 % 54.6 49.4 % 96.7 56.2 % 94.2 32.9 % 35.0 % 31.5 % color in all layers. This may be related to the fact that color is a stronger indicator of the scene type than it is of an object category. Second, while the part of the variance explained by foreground and background color is similar in the fc7 feature of the Places network, it is much larger for the foreground object than for the background object in AlexNet and VGG. Once again, one can hypothesize that it is related to the fact that the color of an object is more informative than the color of its background for object classification. Finally, we note that similarly to our previous experiments, the difference between networks is present in pool5 and increases in the higher layers, indicating that the features become more tuned to the target task in the higher layers of the networks. 4.3. Natural images Embedding. We used ImageNet [28] images to study the embeddings of natural images. Since we have no control over the image content, we cannot perform a detailed analysis of the different factors similar to the previous sections. Our only choice is to consider the images altogether. The direct embedding of natural images is possible but hard to interpret. We can however project the images in the spaces discovered in section 4.2. The resulting embeddings for style and viewpoint are shown in figure 5 and are similar to the embeddings obtained with the CAD models. 2D-3D instance recognition. The observed similarity of the embeddings for natural and rendered images motivates Table 4: Average relative variance over five classes for color/style separation. Foreground/Style Places FG color AlexNet VGG Places Style AlexNet VGG Places ∆L AlexNet VGG pool5 23.4 % 23.2 % 15.0 % 48.9 % 56.6 % 59.0 % 27.7 % 20.3 % 26.0 % fc6 29.4 % 24.0 % 22.6 % 41.3 % 52.1 % 51.4 % 29.3 % 24.0 % 25.9 % fc7 34.9 % 24.0 % 25.0 % 40.3 % 52.5 % 51.3 % 24.8 % 23.5 % 23.6 % Background/Style Places BG color AlexNet VGG Places Style AlexNet VGG Places ∆L AlexNet VGG pool5 24.3 % 17.3 % 9.1 % 51.5 % 63.7 % 71.4 % 24.2 % 19.0 % 19.5 % fc6 29.6 % 16.2 % 13.8 % 40.7 % 59.4 % 64.3 % 29.7 % 24.4 % 22.0 % fc7 35.1 % 14.4 % 14.3 % 40.9 % 61.8 % 65.3 % 24.0 % 23.9 % 20.4 % (a) Car orientation (b) Chair orientation (c) Car Style (d) Chair style (e) 2D-3D retrieval examples an application to retrieve a 3D model of an object present in an image without explicitly performing alignment or detection. The approach is different from approaches in the 2D-3D matching community that find explicit correspondences between the models and the image. We tested this idea using the chair category and computing similarity as dot product between AlexNet features. We rendered 36 azimuth and 4 elevation angles to span typical viewpoints depicted in the natural images. To improve efficiency, we reduced the dimension of our features to 1000 via PCA. This allows us to perform nearest-neighbor retrieval between 1000 natural images and the 144 rendered views of 1261 3D models in approximately 22 seconds total using a Python implementation with pre-computed features. We visualize results in figure 5e and in the supplementary material. We evaluated the viewpoint accuracy using the annotations of [36] and found that the orientation error was below 20 degrees for 60% of the images using pool5 features, 39% using fc6, and 26% using fc7. This is consistent with our earlier finding that orientation is not as well represented in the higher layers. We conducted a user study on Mechanical Turk to evaluate the quality of the style matching for the images where the orientation was correctly estimated with the pool5 features. The workers were presented with a pair of images and asked to judge if the style of the chairs and their orientation were similar or different, similar to [2]. Each pair was evaluated 5 times. There was agreement in 75% of the cases that the Figure 5: PCA embeddings over AlexNet pool5 features for cars and chairs with orientation and style separated. match was fair and in 18% that it was exact. While the above results could probably be beaten by state-of-the-art 2D-3D matching techniques or simply by adding position and scale to our database, they show that our analysis of rendered 3D models is pertinent for understanding the CNN representation of natural images. Real images with pose variation. We applied the methodology of section 4.2 to natural images in the ETH-80 dataset [23]. The dataset spans 8 object categories with 10 instances each captured under 41 viewpoints. As the dataset is significantly smaller than the ModelNet database, this limits considerably the number of variations we can explore. Results using AlexNet features are shown in table 5. The high-level conclusions are the same as those of section 4.2, but the differences are less obvious. The variance is explained more by the style and less by the viewpoint and residual as one progresses toward the higher layers in the network. Please see detailed results for each category in the supplementary material. Table 5: Average relative variance over the 8 categories of the ETH-80 dataset [23]. AlexNet, pool5 AlexNet, fc6 AlexNet, fc7 Rotation 35.4 % 30.2 % 29.5 % Style 21.6 % 27.7 % 30.5 % ∆L 43.0 % 42.0 % 40.0 % 5. Conclusion We have introduced a method to qualitatively and quantitatively analyze deep features by varying the network stimuli according to factors of interest. Utilizing large collections of 3D models, we applied this method to compare the relative importance of different factors and have highlighted the difference in sensitivity between the networks and layers to object style, viewpoint and color. We believe our analysis gives new intuitions and opens new perspectives for the design and use of convolutional neural networks. Acknowledgments. Mathieu Aubry was partly supported by ANR project Semapolis ANR-13-CORD-0003, Intel, a gift from Adobe, and hardware donation from Nvidia. References [1] P. Agrawal, R. Girshick, and J. Malik. Analyzing the performance of multilayer neural networks for object recognition. In ECCV, 2014. 2 [2] M. Aubry, D. Maturana, A. A. Efros, B. C. Russell, and J. Sivic. Seeing 3D chairs: Exemplar part-based 2D-3D alignment using a large dataset of CAD models. In CVPR, 2014. 2, 4, 8 [3] M. Aubry, B. C. Russell, and J. Sivic. Painting-to-3D model alignment via discriminative visual elements. ACM Transactions on Graphics, 33(2), 2014. 3 [4] H. Barrow and J. Tenenbaum. Recovering intrinsic scene characteristics from images. In A. Hanson and E. Riseman, editors, Computer Vision Systems, pages 3–26. Academic Press, N.Y., 1978. 2 [5] P. Berkes and L. Wiskott. On the analysis and interpretation of inhomogeneous quadratic forms as receptive fields. Neural Computation, 2006. 2 [6] I. Biederman. Visual object recognition, volume 2. MIT press Cambridge, 1995. 1 [7] J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE PAMI, 35(8):1872–1886, 2013. 2 [8] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In Proc. BMVC., 2014. 1, 3 [9] B. Cheung, J. Livezey, A. Bansal, and B. Olshausen. Discovering hidden factors of variation in deep networks. In ICLR workshop, 2015. 2 [10] A. Dosovitskiy, J. T. Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. In CVPR, 2015. 2, 4 [11] D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. Technical report, University of Montreal, 2009. 2 [12] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 1, 2 [13] D. Glasner, M. Galun, S. Alpert, R. Basri, and G. Shakhnarovich. Viewpoint-aware object detection and pose estimation. In ICCV, 2011. 4 [14] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006. 2 [15] R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition, 2004. 4 [16] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 3 [17] S. Karayev, M. Trentacoste, H. Han, A. Agarwala, T. Darrell, A. Hertzmann, and H. Winnemöller. Recognizing image style. In Proc. BMVC., 2014. 1 [18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012. 1, 2, 3 [19] T. D. Kulkarni, W. Whitney, P. Kohli, and J. B. Tenenbaum. Deep convolutional inverse graphics network. arXiv preprint arXiv:1503.03167, 2015. 2 [20] K. Lai, L. Bo, X. Ren, and D. Fox. A large-scale hierarchical multi-view RGB-D object dataset. In Proc. Intl. Conf. on Robotics and Automation, 2011. 1 [21] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Comput., 1(4):541–551, 1989. 1 [22] Y. LeCun, F.-J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In CVPR, 2004. 1 [23] B. Leibe and B. Schiele. Analyzing appearance and contour based methods for object categorization. In CVPR, volume 2, pages II–409. IEEE, 2003. 1, 8, 9 [24] A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015. 2 [25] K. Pearson. On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2(11):559–572, 1901. 2 [26] B. Pepik, M. Stark, P. Gehler, and B. Schiele. Teaching 3D geometry to deformable part models. In CVPR, 2012. 4 [27] S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323–2326, 2000. 2 [28] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575, 2014. 3, 7 [29] R. Shepard and J. Metzler. Mental rotation of three dimensional objects. Science, 171(972):701–3, 1971. 6 [30] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR workshop, 2014. 2 [31] K. Tanaka. Neuronal mechanisms of object recognition. Science, 262(5134):685–688, 1993. 1 [32] J. Tenenbaum, V. de Silva, and J. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(550):2319–2323, December 2000. 2 [33] M. Turk and A. Pentland. Eigenfaces for recognition. J. of Cognitive Neuroscience, 3(1):71–86, 1991. 2 [34] C. Vondrick, A. Khosla, T. Malisiewicz, and A. Torralba. HOGgles: Visualizing object detection features. In ICCV, 2013. 2 [35] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D ShapeNets: A deep representation for volumetric shape modeling. In CVPR, 2015. 1, 3 [36] Y. Xiang, R. Mottaghi, and S. Savarese. Beyond PASCAL: A benchmark for 3D object detection in the wild. In WACV, 2014. 8 [37] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014. 2 [38] M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014. 1, 2 [39] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene CNNs. In ICLR, 2015. 1, 2 [40] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using Places database. In NIPS, 2014. 1, 3 Visual Tracking with Fully Convolutional Networks Lijun Wang1,2 , Wanli Ouyang2 , Xiaogang Wang2 , and Huchuan Lu1 1 Dalian University of Technology, China 2 The Chinese University of Hong Kong, Hong Kong, China Abstract We propose a new approach for general object tracking with fully convolutional neural network. Instead of treating convolutional neural network (CNN) as a black-box feature extractor, we conduct in-depth study on the properties of CNN features offline pre-trained on massive image data and classification task on ImageNet. The discoveries motivate the design of our tracking system. It is found that convolutional layers in different levels characterize the target from different perspectives. A top layer encodes more semantic features and serves as a category detector, while a lower layer carries more discriminative information and can better separate the target from distracters with similar appearance. Both layers are jointly used with a switch mechanism during tracking. It is also found that for a tracking target, only a subset of neurons are relevant. A feature map selection method is developed to remove noisy and irrelevant feature maps, which can reduce computation redundancy and improve tracking accuracy. Extensive evaluation on the widely used tracking benchmark [36] shows that the proposed tacker outperforms the state-of-the-art significantly. 1. Introduction Visual tracking, as a fundamental problem in computer vision, has found wide applications. Although much progress [7, 29, 38] has been made in the past decade, tremendous challenges still exist in designing a robust tracker that can well handle significant appearance changes, pose variations, severe occlusions, and background clutters. Existing appearance-based tracking methods adopt either generative or discriminative models to separate the foreground from background and distinct co-occurring objects. One major drawback is that they rely on low-level hand-crafted features which are incapable to capture semantic information of targets, not robust to significant appearance changes, and only have limited discriminative power. Driven by the emergence of large-scale visual data sets and fast development of computation power, Deep (a) (b) (c) Figure 1. Feature maps for target localization. (a)(b) From left to right: input image, the ground truth target heat map, the predicted heat maps using feature maps of conv5-3 and conv4-3 layers of VGG network [27] (See Section 4.2 for the regression method). (c) From left to right: input image, ground truth foreground mask, average feature maps of conv5-3 (top) and conv4-3 (bottom) layers, average selected feature maps conv5-3 (top) and conv4-3 (bottom) layers (See Section 4.1 for the feature map selection method). Neural Networks (DNNs), especially convolutional neural networks [17] (CNNs), with their strong capabilities of learning feature representations, have demonstrated record breaking performance in computer vision tasks, e.g., image classification [16, 27], object detection [8, 23, 22], and saliency detection [33, 39]. Different from hand-crafted features, those learned by CNNs from massive annotated visual data and a large number of object classes (such as ImageNet [4]) carry rich high-level semantic information and are strong at distinguishing objects of different categories. These features have good generalization capability across data sets. Recent studies [28, 1] have also shown that such features are robust to data corruption. Their neuron respons- 3119 es have strong selectiveness on object identities, i.e., for a particular object only a subset of neurons are responded and different objects have different responding neurons. All these motivate us to apply CNNs to address above challenges faced by tracking. Given the limited number of training samples in online tracking and the complexity of deep models, it is inferior to directly apply CNNs to tracking, since the power of CNNs relies on large-scale training. Prior works [6, 35, 12] attempted to transfer offline learned DNN features (e.g. from ImageNet) for online tracking and achieved state-of-the-art performance. However, DNN was treated as a black-box classifier in these works. In contrast, we conduct in-depth study on the properties of CNN features from the perspective of online visual tracking and in order to make better use of them in terms of both efficiency and accuracy. Two such properties are discovered and they motivate the design of our tracking system. First, CNN features at different levels/depths have different properties that fit the tracking problem. A top convolutional layer captures more abstract and high-level semantic features. They are strong at distinguishing objects of different classes and are very robust to deformation and occlusion as shown in Figure 1 (a). However, they are less discriminative to objects of the same category as shown by the examples in Figure 1 (b). A lower layer provides more detailed local features which help to separate the target from distracters (e.g. other objects in the same class) with similar appearance as shown in Figure 1 (b). But they are less robust to dramatic change of appearance, as shown in Figure 1 (a). Based on these observations, we propose to automatically switch the usage of these two layers during tracking depending on the occurrence of distracters. Second, the CNN features pre-trained on ImageNet are for distinguishing generic objects. However, for a particular target, not all the features are useful for robust tracking. Some feature responses may serve as noise. As shown in Figure 1 (c), it is hard to distinguish the target object from background if all the feature maps are used. In contrast, through proper feature selection, the noisy feature maps not related to the representation of the target are cleared out and the remaining ones can more accurately highlight the target and suppress responses from background. We propose a principled method to select discriminative feature maps and discard noisy or unrelated ones for the tracking target. The contributions of this work are three folds: i) We analyze CNN features learned from the large-scale image classification task and find important properties for online tracking. It facilitates further understanding of CNN features and helps to design effective CNN-based trackers. ii) We propose a new tracking method which jointly considers two convolutional layers of different levels so that they complement each other in handling drastic appearance change and distinguishing target object from its similar dis- tracters. This design significantly mitigate drifts. iii) We develop a principled method which automatically selects discriminative feature maps and discards noisy or unrelated ones, further improving tracking accuracy. Evaluation on the widely used tracking benchmark [36] shows that the proposed method well handles a variety of challenging problems and outperforms state-of-the-art methods. 2. Related Work A tracker contains two components: an appearance model updated online and a search strategy to find the most likely target locations. Most recent works [2, 41, 11, 20] focus on the design of appearance models. In generative models, candidates are searched to minimize reconstruction errors. For example, Ross et al. [25] learned subspace online to model target appearance. Recently, sparse coding has been exploited for tracking [21, 3, 32, 31, 30], where the target is reconstructed by a sparse linear combination of target templates. In discriminative models, tracking is cast as a foreground and background separation problem [24, 37, 34]. Online learning algorithms based on CRFs [24], boosting [9], multiple instance learning [2] and structured SVM [10] were applied in tracking and achieved good performance. In [40], the generative and discriminative models were incorporated for more accurate online tracking. All these methods used hand-crafted features. The application of DNNs in online tracking is under fully explored. In [35], a stacked denoising autoencoder (SDAE) was offline trained on an auxiliary tiny image data set to learn generic features and then used for online tracking. In [19], tracking was performed as foreground-background classification with CNN trained online without offline pretraining. Fan et al. [6] used fully convolutional network for human tracking. It took the whole frame as input and predicted the foreground heat map by one-pass forward propagation. Redundant computation was saved. Whereas [35] and [19] operated in a patch-by-by scanning manner. Given N patches cropped from the frame, DNNs had to be evaluated for N times. The overlap between patches leads to a lot of redundant computation. In [12], pre-trained CNN features were used to construct target-specific saliency maps for online tracking. Existing works treated DNNs as black-box feature extractors. Our contributions summarized in Section 1 were not explored in these works. 3. Deep Feature Analysis for Visual Tracking Analysis on deep representations is important to understand the mechanism of deep learning. However, it is still very rare for the purpose of visual tracking. In this section, we present some important properties of CNN features which can better facilitate visual tracking. Our feature analysis is conducted based on the 16-layer VGG network [27] 3120 (a) (b) (c) Figure 2. CNNs trained on image classification task carry spatial configuration information. (a) input image (top) and ground truth foreground mask. (b) feature maps (top row) of conv4-3 layer which are activated within the target region and are discriminative to the background distracter. Their associated saliency maps (bottom row) are mainly focused on the target region. (c) feature maps (top row) of conv5-3 layer which are activated within the target region and capture more semantic information of the category (both the target and background distracter). Their saliency maps (bottom row) present spatial information of the category. Observation 1 Although the receptive field 1 of CNN feature maps is large, the activated feature maps are sparse and localized. The activated regions are highly correlated to the regions of semantic objects . Due to pooling and convolutional layers, the receptive fields of the conv4-3 and conv5-3 layers are very large (92×92 and 196×196 pixels, respectively). Figure 2 shows some feature maps with the maximum activation values in the object region. It can be seen that the feature maps have only small regions with nonzero values. These nonzero values are localized and mainly correspond to the image region of foreground objects. We also use the approach in [26] to obtain the saliency maps of CNN features. The saliency maps in Figure 2 (bottom row) show that the change of input that results in the largest increase of the selected feature maps are located within the object regions. Therefore, the feature maps are capturing the visual representation related to the objects. These evidences indicate that DNN features learned from image classification are localized and correlated to the object visual cues. Thus, these CNN features can be used for target localization. Observation 2 Many CNN feature maps are noisy or unrelated for the task of discriminating a particular target from its background. The CNN features pre-trained on ImageNet can describe a large variety of generic objects and therefore they are sup1 We use the term receptive field to denote the input image region that are connected to a particular neuron in the feature maps 500 350 300 Number of Feature Maps Number of Feature Maps pre-trained on the ImageNet image classification task [4], which consists of 13 convolutional layers followed by 3 fully connected layers. We mainly focus on the conv4-3 layer (the 10-th convolutional layer) and the conv5-3 layer (the 13-th convolutional layer), both of which generate 512 feature maps. 250 200 150 100 50 0 0 5,000 10,000 15,000 Activation Value 20,000 25,000 400 300 200 100 0 0 200 400 600 800 1000 1200 1400 1600 1800 Activation Value Figure 3. Activation value histograms of feature maps in conv4-3 (left) and conv5-3 (right). posed to detect abundant visual patterns with a large number of neurons. However, when tracking a particular target object, it should focuses on a much smaller subset of visual patterns which well separate the target from background. As illustrated in Figure 1 (c), the average of all the feature maps is cluttered with background noise. And we should discard feature maps that have high response in both target region and background region so that the tracker does not drift to the background regions. Figure 3 shows the histograms of the activation values for all the feature maps within the object region. The activation value of a feature map is defined as the sum of its responses in the object region. As demonstrated in Figure 3, most of the feature maps have small or zero values within the object region. Therefore, there are lots of feature maps that are not related to the target object. This property provides us the possibility in selecting only a small number of feature maps with small degradation in tracking performance. Observation 3 Different layers encode different types of features. Higher layers capture semantic concepts on object categories, whereas lower layers encode more discriminative features to capture intra class variations. Because of the redundancy of feature maps, we employ a sparse representation scheme to facilitate better visualization. By feeding forward an image of an object through the VGG network, we obtain the feature maps F ∈ Rd×n of a convolutional layer (either the conv4-3 or conv5-3 layer), 3121 Table 1. Face classification accuracy using different feature maps. Experiment 1 is to classify face and non-face. Experiment 2 is classify face identities. Feature map conv4-3 conv5-3 (a) (b) Figure 4. The first and the third rows are input images. The second and the fourth rows are reconstructed foreground masks using conv5-3 feature maps. The sparse coefficients are computed using the images in the first column and directly applied to the other columns without change. where each feature map is reshaped into a d-dimensional vector and n denotes the number of feature maps. We further associate the image with a foreground mask π ∈ Rd×1 , where the i-th element πi = 1 if the i-th neuron of each feature map is located within the foreground object, and πi = 0, otherwise. We reconstruct the foreground mask using a subset of the feature maps by solving min kπ − Fck22 + λkck1 , c s.t. c  0, (1) where c ∈ Rn×1 is the sparse coefficient vector, and λ the parameter to balance the reconstruction error and sparsity. Figure 4 shows some of the reconstructed foreground masks using the feature maps of conv5-3. For the two examples (face and motorcycle) in Figure 4, we only compute the sparse coefficients for images in the first column and use the coefficients to reconstruct foreground masks for the rest of the columns. The selected feature maps in Figure 4 (a) capture the semantic concepts of human faces and are robust to faces with appearance variation and even identity change. The selected feature maps in Figure 4 (b) accurately separate the target from cluttered background and are invariant to pose variation and rotation. Although trained on the image classification task, the high-level semantic representation of object categories encoded by the conv5-3 layer enables object localization. However, these features are not discriminative enough to different objects of the same category, thus they can not be directly applied to visual tracking. Compared with the conv5-3 feature maps, the features captured by conv4-3 are more sensitive to intra-class appearance variation. In Figure 2, the selected feature maps of conv4-3 can well separate the target person from the other non-target person. Besides, different feature maps focus Experiment 1 76.42% 89.56% Experiment 2 83.83% 57.28% on different object parts. To further verify this, we conduct two quantitative experiments. 1800 human face images belonging to six identities and 2000 images containing non-face objects are collected from the benchmark sequences [36]. Each image is associated with a foreground mask to indicate the region of the foreground object. In the first experiment, we evaluate the accuracy in classifying the images into face and non-face using the conv4-3 and conv5-3 layers separately. Three face images belonging to three identities are selected as positive training samples to compute a set of sparse coefficients {c1 , c2 , c2 } via (1). At the test stage, given the feature maps F and the foreground mask π of an input image, the reconstruction error e for the foreground map is computed by e = min kπ − Fci k22 . i (2) The image is classified as a face image if its reconstruction error e is less than a predefined threshold. Otherwise, it is classified as a non-face image. In the second experiment, our task is to classify all the face images into different identities. For each identity, 20 images are used as the training samples to learn the sparse coefficients ci , i = 1, 2, . . . , 6 using (1). At the test stage, the foreground mask reconstruction error for each identity is calculated, and the test image is classified as the identity that has the minimum error as follows: id = arg min kπ − Fci k22 . i (3) The classification accuracy using the feature maps of conv4-3 and conv5-3 for the two experiments are demonstrated in Table 1. The feature maps of conv5-3 encode high level semantic information and can better separate face from non-face objects. But they achieve lower accuracy than the features maps of conv4-3 in discriminating one identity from another. The feature maps of conv4-3 preserve more middle-level information and enables more accurate classification of different images belonging to the same category (human faces). But they are worse than the feature maps of conv5-3 in discriminating face from non-face. These results motivate us to consider these two layers jointly for more robust tracking. 4. Proposed Algorithm An overview (Figure 5) of the proposed fully convolutional network based tracker (FCNT) is as follows: 3122 SNet Feature Map Selection VGG Net … (b) Conv4-3 (c) … Distracter Detection GNet Conv5-3 (a) (e) Feature Map Selection (d) Figure 5. Pipeline of our algorithm. (a) Input ROI region. (b) VGG network. (c) SNet. (d) GNet. (e) Tracking results. 1. For a given target, a feature map selection process is performed on the conv4-3 and conv5-3 layers of the VGG network to select the most relevant feature maps and avoid overfitting on noisy ones. 2. A general network (GNet) that captures the category information of the target is built on top of the selected feature maps of the conv5-3 layer. 3. A specific network (SNet) that discriminates the target from background with similar appearance is built on top of the selected feature maps of the conv4-3 layer. 4. Both GNet and SNet are initialized in the first frame to perform foreground heat map regression for the target and adopt different online update strategies. 5. For a new frame, a region of interest (ROI) centered at the last target location containing both target and background context is cropped and propagated through the fully convolutional network. 6. Two foreground heat maps are generated by GNet and SNet, respectively. Target localization is performed independently based on the two heat maps. 7. The final target is determined by a distracter detection scheme that decides which heat map in step 6 to be used. 4.1. Feature Map Selection The proposed feature map selection method is based on a target heat map regression model, named as sel-CNN, and is conducted independently on the conv4-3 and conv5-3 layers of VGG. The sel-CNN model consists of a dropout layer followed by a convolutional layer without any nonlinear transformation. It takes the feature maps (conv4-3 or con53) to be selected as input to predict the target heat map M, which is a 2-dimensional Gaussian centered at the ground truth target location with variance proportional to the target size (See Figure 1 (a) and (b) for an example). The model is trained by minimizing the square loss between the predicted foreground heat map M̂ and the target heat map M: Lsel = kM̂ − Mk2 . (4) After parameter learning using back-propagation converges, we fix the model parameters and select the feature maps according to their impacts on the loss function. The input feature maps F are vectorized into a vector denoted by vec(F). Denote fi as the i-th element of vec(F). The change of the loss function caused by the perturbation of the feature map δF can be computed by a two-order Taylor expansion as follows: δLsel = X i gi δfi + 1X 1X hij δfi δfj , hii (δfi )2 + 2 i 2 i6=j (5) ∂ 2 Lsel sel where gi = ∂L and h = are, respectively, the ij ∂fi ∂fi ∂fj first and second order derivatives of the objective function with respect to the input feature maps. The number of elements in the feature maps is very large (> 270, 000). The complexity for computing all the second order derivatives hij is O(270, 0002 ), which is too time consuming. We approximate the Hessian matrix with a diagonal matrix, in which the third term of the right hand side of (5) is neglected. Both the first derivatives gi and the second derivatives hii can be efficiently computed via back-propagation. We define the significance of the element fi as the change of the objective function after setting fi to zero, i.e., δfi = 0 − fi . According to (5), the significance of fi can then be computed as 1 si = −gi fi + hii fi2 . 2 (6) The significance of the k-th feature map are further defined asPthe summation of significance of all its elements Sk = x,y s(x, y, k), where s(x, y, k) is the significance of the element indexed by location (x, y) on the k-th feature map. All the feature maps are sorted in the descending order by their significance, and the top K feature maps are selected. These selected feature maps have significant impact on the objective function and thus are most relevant to the tracking task. Our feature map selection method can be conducted in an online fashion. In our experiments, we only conduct feature selection at the first frame and have achieved good performance. This should be partially attributed to the robustness of CNN features. 3123 The idea of using quadratic approximation of the cost function to remove connections in networks can be traced back to 1989 [18]. The aim was to reduce the number of parameters and improve speed, while we target on removing noisy feature maps and improving tracking accuracy. 4.2. Target Localization Figure 5 (c) and (d) show the design of the CNNs for target localization. After feature map selection in the first frame, we build the SNet and the GNet on top of the selected conv4-3 and conv5-3 feature maps, respectively. Both networks share the same architecture that consists of two additional convolutional layers. The first additional convolutional layer has convolutional kernels of size 9×9 and outputs 36 feature maps as the input to the next layer. The second additional convolutional layer has kernels of size 5 × 5 and outputs the foreground heat map of the input image. ReLU is chosen as the nonlinearity for these two layers. SNet and GNet are initialized in the first frame by minimizing the following square loss function: L = LS + LG , LU = kM̂U − Mk2F + βkWU k2F , (7) where the subscript U ∈ {S, G} indicates SNet and GNet, respectively; M̂U represents the foreground heat map predicted by the network; M is the target heat map, WU is the weight parameter of the convolutional layers; β is a trade off parameter for weight decay. Note that the sel-CNN for selecting features and the SNet and GNet for localization are different in CNN structures. The sel-CNN architecture is very simple to avoid using noisy feature maps to overfit the objective function, whereas the SNet and GNet are more complex. Since the noisy feature maps have been discarded by the feature map selection, more complex models facilitate more accurate tracking. Detailed experimental results and analysis on the selection of different model architectures for localization and feature map selection are provided in the supplementary materials. In a new frame, we crop a rectangle ROI region centered at the last target location. By forward propagating the ROI region through the networks, the foreground heat maps are predicted by both GNet and SNet. Target localization is first performed on the heat map produced by GNet. Denote the target location as X̂ = (x, y, σ) , where x, y and σ represent the center coordinates and scale of the target bounding box, respectively. Given the target location X̂t−1 in the last frame, we assume the locations of target candidates in the current frame are subject to a Gaussian distribution p(Xt |X̂t−1 ) = N (Xt ; X̂t−1 , Σ), (8) where Σ is a diagonal covariance matrix that indicates the variances of the location parameters. The confidence of the i-th candidate is computed as the summation of allPthe heat map values within the candidate region conf i = j∈Ri M̂G (j), where M̂G denotes the heat map generated by GNet; Ri is the region of the i-th target candidate according to its location parameter Xti (8); j denotes the coordinate index. The candidate with the highest confidence is predicted as the target by GNet. According to the analysis in Section 3, GNet based on the conv5-3 layer captures semantic features and is highly invariant to intra class variation. Hence, the foreground heat map generated by GNet highlights both the target and background distracters with similar appearances. To prevent the tracker from drifting to background, we further promote a distracter detection scheme to determine the final target location. Denote the target location predicted by GNet as X̂G , the corresponding target region in the heat map as RG . The probability of distracter occurring in background is evaluated by the proportion between the confidence values outside and inside the target region P j∈M̂ −R M̂G (j) Pd = P G G , (9) k∈RG M̂G (k) where M̂G − RG represents the background region on heat map M̂G . When the proportion Pd is less than a threshold (0.2 in all the experiments), we assume no co-occurring distracter and use the target location predicted by GNet as the final result. Otherwise, the same target localization procedure described above is performed on the heat map M̂S predicted by SNet to determine the final target location. 4.3. Online Update To avoid the background noise introduced by online update, we fix GNet and only update SNet after the initialization in the first frame. SNet is updated following two different rules: the adaptation rule and the discrimination rule, which aim to adapt SNet to target appearance variation and improve the discriminative power for foreground and background, respectively. According to the adaptation rule, we finetune SNet every 20 frames using the most confident tracking result within the intervening frames. Based on the discrimination rule, when distracters are detected using (9), SNet is further updated using the tracking results in the first frame and the current frame by minimizing min βkWS k2F + X h M̂1S (x, y) − M1 (x, y) x,y + 1 − Φt (x, y)  h M̂tS (x, y) − Mt (x, y) i2 i2  , (10) where WS denotes the convolutional weight of SNet; (x, y) are spatial coordinates; M̂tS and Mt represent the heat map for the t-th frame predicted by SNet and the heat map 3124 Success plots of OPE 1 0.9 0.9 0.8 0.8 0.7 0.7 Success rate Precision Precision plots of OPE 1 0.6 0.5 FCNT [0.856] MEEM [0.828] TGPR [0.766] KCF [0.740] Struck [0.656] SCM [0.649] TLD [0.608] DLT [0.587] CXT [0.575] ASLA [0.532] 0.4 0.3 0.2 0.1 0 0 10 20 30 40 0.6 0.5 0.4 0.3 0.2 0.1 0 0 50 FCNT [0.599] MEEM [0.567] TGPR [0.529] KCF [0.514] SCM [0.499] Struck [0.474] TLD [0.437] DLT [0.436] ASLA [0.434] CXT [0.426] 0.2 Location error threshold 0.4 0.6 0.8 1 Overlap threshold Figure 6. The precision plots and success plots of OPE for the top 10 trackers. The performance score for each tracker is shown in the legend. The performance score of precession plot is at error threshold of 20 pixels while the performance score of success plot is the AUC value. Table 2. Average precision scores on different attributes: illumination variation (IV), out-of-plane rotation (OPR), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-view (OV), background cluttered (BC) and low resolution (LR). The best and the second best results are in red and green colors, respectively. IV OPR SV OCC DEF MB FM IPR OV BC LR Overall SCM 0.594 0.618 0.672 0.640 0.586 0.339 0.333 0.597 0.429 0.578 0.305 0.649 Struck 0.558 0.597 0.639 0.564 0.521 0.551 0.604 0.617 0.539 0.585 0.545 0.656 TLD 0.537 0.596 0.606 0.563 0.512 0.518 0.551 0.584 0.576 0.428 0.349 0.608 DLT 0.534 0.561 0.590 0.574 0.563 0.453 0.446 0.548 0.444 0.495 0.396 0.587 ASLA 0.517 0.518 0.552 0.460 0.445 0.278 0.253 0.511 0.333 0.496 0.156 0.532 CXT 0.501 0.574 0.550 0.491 0.422 0.509 0.515 0.610 0.510 0.443 0.371 0.575 MEEM 0.778 0.838 0.787 0.801 0.859 0.740 0.757 0.790 0.730 0.807 0.494 0.828 KCF 0.728 0.729 0.679 0.749 0.740 0.650 0.602 0.725 0.650 0.753 0.381 0.740 generated according to the predicted target location (a 2dimensional Gaussian centered at the target location), respectively; the foreground mask Φt indicates the predicted target bounding box, i.e., Φt (x, y) = 1 if the location (x, y) belongs to the target region and Φt (x, y) = 0, otherwise. The second term in (10) corresponds to loss for locating the target in the first frame. When distracters appear or the target undergoes severe occlusion in the current frame, the estimated target region is not reliable for learning target appearance. Therefore, we choose a conservative scheme by adding the first frame to supervise update so that the learned model still captures the appearance in the first frame. Meanwhile, the third term in (10) removes the loss in the unreliable target region and only considers those within the background region in the current frame. It enforces the model to put more efforts on assigning the co-occurring distracters as background. The combination of the second term and the third term in (10) can help SNet to better separate the target from background and alleviate the model degradation caused by occlusion or distracters. 5. Experiments Setup. The proposed FCNT tracker is implemented in MATLAB based on the wrapper of Caffe framework [14], and runs at 3 fps on a PC with a 3.4GHz CPU and a TI- TGPR 0.687 0.741 0.703 0.708 0.768 0.578 0.575 0.705 0.576 0.761 0.539 0.766 FCNT 0.830 0.831 0.830 0.797 0.917 0.789 0.767 0.811 0.741 0.799 0.765 0.856 FCNTG 0.776 0.820 0.806 0.745 0.917 0.666 0.686 0.797 0.636 0.722 0.577 0.794 FCNTS 0.731 0.775 0.762 0.764 0.884 0.737 0.727 0.749 0.641 0.679 0.633 0.801 FCNT512 0.791 0.789 0.799 0.740 0.919 0.734 0.725 0.778 0.648 0.722 0.747 0.824 FCNT256 0.758 0.779 0.787 0.730 0.857 0.712 0.671 0.824 0.520 0.741 0.740 0.817 FCNT128 0.760 0.758 0.755 0.740 0.896 0.743 0.731 0.750 0.650 0.674 0.761 0.800 TAN GPU. The source code is publicly available2 . Both the sel-CNN for feature map selection and the GNet and SNet for target localization are trained in the first frame using back-propagation for 50 iterations. Afterwards, SNet are finetuned for 3 iterations at each update step. The learning rates are set to 1e − 9 for the sel-CNN and 1e − 7 for the GNet and SNet. The number of feature maps selected by the proposed feature selection method is set to K = 384 for both the conv4-3 and conv5-3 layers. The size of the input ROI region centered at target location is 386 × 386 pixels. The weight decay parameter β in (7) and (10) is set to 0.005. At each frame, 600 target candidates are randomly sampled. The variance of location parameters in (8) are set to {10, 10, 0.004} for x, y translation and scale, respectively. All the parameters are fixed through the experiment. Evaluation Methodology. We evaluate the proposed FCNT tracker on the benchmark data set [36] which includes 50 sequences and the results of 29 trackers. In addition, we also compare our method with three recent state-of-the-art methods MEEM [38], TGPR [7] and KCF [11] for a more thorough comparison. The sequences are further tagged with 11 attributes according to different challenging factors. We use the precision plot and the success plot to evaluate all the trackers. The precision plot demonstrates the per2 http://ice.dlut.edu.cn/lu/index.html 3125 FCNT64 0.787 0.751 0.785 0.707 0.850 0.729 0.713 0.751 0.554 0.727 0.750 0.798 Table 3. Average success scores on different attributes: illumination variation (IV), out-of-plane rotation (OPR), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-view (OV), background cluttered (BC) and low resolution (LR).The best and the second best results are in red and green colors, respectively. IV OPR SV OCC DEF MB FM IPR OV BC LR Overall SCM 0.473 0.470 0.518 0.487 0.448 0.293 0.296 0.458 0.361 0.450 0.279 0.499 Struck 0.428 0.432 0.425 0.413 0.393 0.433 0.462 0.444 0.459 0.458 0.372 0.474 TLD 0.399 0.420 0.421 0.402 0.378 0.404 0.417 0.416 0.457 0.345 0.309 0.437 DLT 0.405 0.412 0.455 0.423 0.394 0.363 0.360 0.411 0.367 0.339 0.346 0.436 ASLA 0.429 0.422 0.452 0.376 0.372 0.258 0.247 0.425 0.312 0.408 0.157 0.434 CXT 0.368 0.418 0.389 0.372 0.324 0.369 0.388 0.452 0.427 0.338 0.312 0.426 MEEM 0.548 0.562 0.503 0.559 0.582 0.565 0.568 0.526 0.597 0.574 0.367 0.567 KCF 0.493 0.495 0.427 0.514 0.534 0.497 0.459 0.497 0.550 0.535 0.312 0.514 centage of frames where the distance between the predicted target location and the ground truth location is within a given threshold. All the trackers are ranked according to the precision scores at the threshold of 20 pixels. Whereas the success plot illustrates the percentage of frames where the overlap ratio between the predicted bounding box and the ground truth bounding box is higher than a threshold τ ∈ [0, 1]. The area under curve (AUC) of each success plot are used to rank the tracking algorithms. We report the results of one pass evaluation (OPE) [36] for the proposed FCNT tracker and the top 10 algorithms in each plot, including MEEM [38], TGPR [7], KCF [11], SCM [40], Struck [10], TLD [15], DLT [35], ASLA [13] and CXT [5]. Results. The precision plot and the success plot of the compared trackers on 50 sequences are demonstrated in Figure 6. The proposed FCNT tracker outperforms all the other trackers in terms of both average precision score and success score. To facilitate better analysis on the tracking performance, we further evaluate all the trackers on sequences with 11 attributes. Table 2 and Table 3 demonstrate that the proposed FCNT can well handle a variety of challenging factors and consistently outperform state-of-the-art methods in almost all the challenges. To demonstrate the robustness of the features learned by DNNs from large scale image classification, we also compare with [19] which use a convolutional network for tracking without pre-training. On the 16 adopted test sequences, the proposed FCNT achieves a precision score of 0.88 and a success score of 0.85, whereas the reported results in [19] are 0.83 and 0.83, respectively. Our pre-trained network outperforms the method in [19] with a large margin. We also note that the proposed FCNT tracker has some failure cases in handling the low resolution (LR) challenge and achieves a relatively lower success score on the LR attribute (Table 3). One reason is that the VGG network is pre-trained on the Imagenet data set with training images of high resolutions. Ablation Study. To further investigate the effectiveness of tracking by considering multiple layers and the proposed feature map selection method, we report the performance of different variants of the proposed algorithm in Table 2 and TGPR 0.486 0.507 0.443 0.494 0.556 0.440 0.441 0.487 0.431 0.543 0.351 0.529 FCNT 0.598 0.581 0.558 0.571 0.644 0.580 0.565 0.555 0.592 0.564 0.514 0.599 FCNTG 0.519 0.532 0.507 0.491 0.605 0.452 0.496 0.510 0.478 0.494 0.416 0.524 FCNTS 0.546 0.548 0.527 0.557 0.638 0.563 0.545 0.519 0.498 0.510 0.452 0.574 FCNT512 0.565 0.550 0.531 0.528 0.649 0.524 0.523 0.532 0.502 0.514 0.495 0.575 FCNT256 0.546 0.544 0.525 0.519 0.618 0.518 0.493 0.561 0.419 0.527 0.497 0.570 FCNT128 0.558 0.539 0.515 0.540 0.643 0.539 0.534 0.520 0.514 0.498 0.520 0.568 3, where FCNTG and FCNTS denote the proposed method using only GNet and SNet for tracking; FCNTK represents the proposed method using K selected feature maps from both the cnov4-3 and conv5-3 layers of VGG network, and no feature map selection is conducted for K = 512. FCNTS exploits more discriminative feature and with proper online update it has higher performance than FCNTG . By considering both the higher layer and the middle layer, the proposed FCNT can better deal with different challenging factors and outperform both FCNTG and FCNTS which only use the feature maps of a single layer. The proposed FCNT tracker uses less feature maps (386) achieves higher performance than FCNT512 . The FCNT64 tracker uses only 1/8 of all the feature maps and can still compare favorably against state-of-the-art methods. This further demonstrate that the proposed feature map selection method can effectively remove unreliable and noisy feature maps and retain relevant ones, which effectively avoids overfitting and improves training convergence rate. 6. Conclusion In this paper, we empirically present some important properties of CNN features under the viewpoint of visual tracking. Based on these properties, we propose a tracking algorithm using fully convolutional networks pre-trained on image classification task. We observe that convolutional layers at different levels have different properties. And we jointly consider these properties in order to capture the semantic information of the target and discriminate the target from background distracters. We further develop a principled feature map selection method to select discriminative features and discard noisy or unrelated ones. Our approach is shown to effectively improve the tracking performance on challenging scenarios. Acknowledgements. This work is supported by the Natural Science Foundation of China (NSFC) #61472060, the Fundamental Research Funds for the Central Universities under Grant DUT14YQ101, Hong Kong Innovation and Technology Support Programme (ITS/221/13FP), and the Research Grants Council of Hong Kong (CUHK 417011, CUHK 419412, CUHK 14207814). 3126 FCNT64 0.565 0.529 0.519 0.515 0.614 0.532 0.513 0.520 0.430 0.517 0.483 0.559 References [1] P. Agrawal, R. B. Girshick, and J. Malik. Analyzing the performance of multilayer neural networks for object recognition. In ECCV, 2014. 1 [2] B. Babenko, M.-H. Yang, and S. Belongie. Robust object tracking with online multiple instance learning. TPAMI, 33(8):1619–1632, 2011. 2 [3] C. Bao, Y. Wu, H. Ling, and H. Ji. Real time robust l1 tracker using accelerated proximal gradient approach. In CVPR, 2012. 2 [4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 1, 3 [5] T. B. Dinh, N. Vo, and G. Medioni. Context tracker: Exploring supporters and distracters in unconstrained environments. In CVPR, 2011. 8 [6] J. Fan, W. Xu, Y. Wu, and Y. Gong. Human tracking using convolutional neural networks. TNN, 21(10):1610–1623, 2010. 2 [7] J. Gao, H. Ling, W. Hu, and J. Xing. Transfer learning based visual tracking with gaussian processes regression. In ECCV. 2014. 1, 7, 8 [8] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 1 [9] H. Grabner, C. Leistner, and H. Bischof. Semi-supervised on-line boosting for robust tracking. In ECCV, 2008. 2 [10] S. Hare, A. Saffari, and P. H. Torr. Struck: Structured output tracking with kernels. In ICCV, 2011. 2, 8 [11] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. Highspeed tracking with kernelized correlation filters. TPAMI, 37(3):583–596, 2015. 2, 7, 8 [12] S. Hong, T. You, S. Kwak, and B. Han. Online tracking by learning discriminative saliency map with convolutional neural network. In ICML, 2015. 2 [13] X. Jia, H. Lu, and M.-H. Yang. Visual tracking via adaptive structural local sparse appearance model. In CVPR, 2012. 8 [14] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM Multimedia, pages 675–678, 2014. 7 [15] Z. Kalal, K. Mikolajczyk, and J. Matas. Tracking-learningdetection. TPAMI, 34(7):1409–1422, 2012. 8 [16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 1 [17] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 1 [18] Y. LeCun, J. S. Denker, S. A. Solla, R. E. Howard, and L. D. Jackel. Optimal brain damage. In NIPS, 1989. 6 [19] H. Li, Y. Li, and F. M. Porikli. Robust online visual tracking with a single convolutional neural network. In ACCV, 2014. 2, 8 [20] X. Li, Z. Han, L. Wang, and H. Lu. Visual tracking via random walks on graph model. IEEE Transactions on Cybernetics, PP(99):1–1, 2015. 2 [21] X. Mei and H. Ling. Robust visual tracking using l1 minimization. In CVPR, 2009. 2 [22] W. Ouyang and X. Wang. Joint deep learning for pedestrian detection. In ICCV, 2013. 1 [23] W. Ouyang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, S. Yang, Z. Wang, C.-C. Loy, et al. Deepid-net: Deformable deep convolutional neural networks for object detection. In CVPR, 2015. 1 [24] X. Ren and J. Malik. Tracking as repeated figure/ground segmentation. In CVPR, 2007. 2 [25] D. A. Ross, J. Lim, R.-S. Lin, and M.-H. Yang. Incremental learning for robust visual tracking. IJCV, 77(1-3):125–141, 2008. 2 [26] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013. 3 [27] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 1, 2 [28] Y. Sun, X. Wang, and X. Tang. Deeply learned face representations are sparse, selective, and robust. CoRR, abs/1412.1265, 2014. 1 [29] D. Wang and H. Lu. Visual tracking via probability continuous outlier model. In CVPR, 2014. 1 [30] D. Wang, H. Lu, Z. Xiao, and M.-H. Yang. Inverse sparse tracker with a locally weighted distance metric. TIP, 24(9):2646–2657, 2015. 2 [31] D. Wang, H. Lu, and M. Yang. Robust visual tracking via least soft-threshold squares. TCSVT, PP(99):1–1, 2015. 2 [32] D. Wang, H. Lu, and M.-H. Yang. Online object tracking with sparse prototypes. TIP, 22(1):314–325, 2013. 2 [33] L. Wang, H. Lu, X. Ruan, and M.-H. Yang. Deep networks for saliency detection via local estimation and global search. In CVPR, 2015. 1 [34] L. Wang, H. Lu, and D. Wang. Visual tracking via structure constrained grouping. Signal Processing Letters, 22(7):794– 798, 2015. 2 [35] N. Wang and D. Yeung. Learning a deep compact image representation for visual tracking. In NIPS, 2013. 2, 8 [36] Y. Wu, J. Lim, and M.-H. Yang. Online object tracking: A benchmark. In CVPR, 2013. 1, 2, 4, 7, 8 [37] F. Yang, H. Lu, and M.-H. Yang. Robust superpixel tracking. TIP, 23(4):1639–1651, 2014. 2 [38] J. Zhang, S. Ma, and S. Sclaroff. Meem: Robust tracking via multiple experts using entropy minimization. In ECCV. 2014. 1, 7, 8 [39] R. Zhao, W. Ouyang, H. Li, and X. Wang. Saliency detection by multi-context deep learning. In CVPR, 2015. 1 [40] W. Zhong, H. Lu, and M. Yang. Robust object tracking via sparse collaborative appearance model. TIP, 23(5):2356– 2368, 2014. 2, 8 [41] B. Zhuang, H. Lu, Z. Xiao, and D. Wang. Visual tracking via discriminative sparse similarity map. TIP, 23(4):1872–1881, 2014. 2 3127