A Unified Batch Selection Policy for Active Metric Learning Priyadarshini K1 , Siddhartha Chaudhuri2 , Vivek Borkar1 , and Subhasis Chaudhuri1 1 IIT Bombay Adobe Research priyadarshini.k@iitb.ac.in,sidch@adobe.com,{borkar,sc}@ee.iitb.ac.in arXiv:2102.07365v5 [cs.LG] 1 Aug 2021 2 Abstract. Active metric learning is the problem of incrementally selecting highutility batches of training data (typically, ordered triplets) to annotate, in order to progressively improve a learned model of a metric over some input domain as rapidly as possible. Standard approaches, which independently assess the informativeness of each triplet in a batch, are susceptible to highly correlated batches with many redundant triplets and hence low overall utility. While a recent work [20] proposes batch-decorrelation strategies for metric learning, they rely on ad hoc heuristics to estimate the correlation between two triplets at a time. We present a novel batch active metric learning method that leverages the Maximum Entropy Principle to learn the least biased estimate of triplet distribution for a given set of prior constraints. To avoid redundancy between triplets, our method collectively selects batches with maximum joint entropy, which simultaneously captures both informativeness and diversity. We take advantage of the submodularity of the joint entropy function to construct a tractable solution using an efficient greedy
algorithm based on Gram-Schmidt orthogonalization that is provably 1 − 1e -optimal. Our approach is the first batch active metric learning method to define a unified score that balances informativeness and diversity for an entire batch of triplets. Experiments with several real-world datasets demonstrate that our algorithm is robust, generalizes well to different applications and input modalities, and consistently outperforms the state-of-the-art. Keywords: Batch active learning · Perceptual metric · Submodular optimization · Maximum Entropy Principle. 1 Introduction Understanding similarity between two objects is fundamental to many vision and machine learning tasks, e.g. object retrieval [33], clustering [35] and classification [30]. Most existing methods model a discrete measure of similarity based on class labels: all inter-class samples are considered equally dissimilar, even though their features differ by different degrees. But human estimation of perceptual (dis)similarity is often more fine-grained. We may choose, for example, continuous measures such as the degree of perceived similarity in taste or visual appearance for comparing two food dishes, rather than discrete categorical labels (Figure 1). Thus, it is important to build a continuous perceptual space to model human-perceived similarity between objects. Recent studies 2 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri Fig. 1: Difference between class-based and perceptual distances on two different types of triplets. In each case, the class-based metric dC fails to capture intra-class variations and inter-class similarities and is not compatible with the perceptual metric dH . demonstrate the importance of perceptual metrics in several tasks in computer vision and cognitive science [36,15,19]. Early work on perceptual metric learning focuses on non-parametric methods (e.g. Multidimensional scaling (MDS) [18]) which use numerical measurements of pairwise similarity for training. These are hard to gather and suffer from inconsistency. Instead, similarity comparisons of the form “Is object xi more similar to object xj than object xk ?” are easier to gather and more stable [14]. They form a useful foundation for several tasks, including perceptual metric learning. However, the number of possible triplets of n objects is O(n3 ), making it infeasible to label even a significant fraction of them. Fortunately, many triplets are redundant and we can effectively model the metric using only a few high-utility triplets (Figure 2). Thus it is imperative to identify and annotate a subset of high-quality triplets that are jointly informative for the model, without knowing the annotations of any triplets in advance. We stress this last point since it renders common triplet sampling strategies such as (semi-)hard negative mining, which rely on access to a fully annotated dataset, inadmissible. Results of our approach after annotating 18% triplets Resuls of our aproach after annotating 13% triplets Query Query Resuls of random sampling after annotating 13% triplets Results of random sampling after annotating 18% triplets Fig. 2: Top-3 retrieved images ranked from most to least similar by a perceptual metric (visual appearance for birds, and taste for food) trained on randomly selected (but correctly annotated) triplets vs. high-quality triplets identified for annotation by our method. For a fair comparison, both methods run for equal training rounds and solicit annotations for equal amounts of training data – 13% of the CUB-200 bird dataset on the left, and 18% of the Yummly-Food dataset on the right. A Unified Batch Selection Policy for Active Metric Learning 3 Active learning is a standard technique that addresses this issue by iteratively identifying small batches of informative samples and soliciting labels for them. While extensively studied for class label-based learning tasks, there exists very little literature [31,9,20] on active learning which focuses on perceptual/general metric learning. Further, these works merely assess the informativeness of individual triplets with uncertainty measures, which assume a triplet with high prediction uncertainty is more crucial to label. Although effective in many scenarios, such an uncertainty measure makes a myopic decision based solely on the current model’s prediction and fails to capture the triplets’ collective distribution as a whole. Independently assessed triplets may themselves have much redundancy even if they are individually informative. Hence the triplets should be not merely informative but also diverse or decorrelated. Kumari et al. [20] proposed a method for selecting informative and decorrelated batches of triplets for active metric learning. However, their approach suffers from three major limitations: (1) The active learning strategy is based on a two stage optimization for informativeness (choice of an overcomplete batchpool of individually informative triplets) and diversity (subsequent trimming of the batchpool), applied sequentially. It does not always ensure an optimal tradeoff between the two criteria. (2) The proposed diversity measures are all ad-hoc with no principled connection to informativeness. Being heuristic, no single measure works consistently well in all cases, making it harder for a user to select which measure to use in practice. (3) The informativeness of a triplet is determined using a point estimate of the perceptual metric. Bias in the latter, e.g., because of suboptimal batch selection in prior iterations, directly translates to bias in informativeness, which can misguide the strategy. To mitigate these issues, we propose a new batch active learning algorithm developed specifically for triplet-based metric learning. Our key insight is to express a set of (unannotated) triplets as a vector of random variables, and select batches of triplets that maximize the joint entropy measure. Thus, instead of separately expressing and optimizing informativeness for individual triplets and diversity for pairs of triplets, we develop a single probabilistic informativeness measure for a batch of triplets. We also provide computationally efficient approximate solutions with provable guarantees. Specifically, our main technical contributions are: 1. We propose to use the joint entropy of the distribution of triplet margins to rank a batch of unannotated triplets. We estimate the second-order statistics (mean and covariance) of triplet margins by randomly perturbing the current model trained on prior batches as in [7], to characterize the distribution. 2. Using the Maximum Entropy Principle, we arrive at a Gaussian distribution compatible with the given empirical mean and covariance, whose entropy is characterized by the determinant of the covariance matrix.
As exact maximization of the joint entropy is prohibitively expensive (there are m b possible batches of size b from m triplets), we use the fact that entropy is monotone increasing and submodular to justify a greedy policy which is provably (1 − 1e )-optimal [22]. 3. We achieve further computational efficiency by using the fact that the covariance matrix is a Gram matrix, and its determinant can be computed using efficient recursion. Our method recursively maximizes successive projection errors of a set of vectors, picked one at a time, when projected onto the span of previous choices. 4 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri This amounts to successive maximization of the conditional entropy, and is easily implemented using Gram-Schmidt orthogonalization. We demonstrate the effectiveness of our approach through extensive experiments on different applications and data in different modalities (image, taste and haptic). In addition to having a sound theoretical justification, our method provides a significant performance gain over the current state-of-the-art. 2 Related Work The prior work can be roughly divided into three categories. We review representative techniques in each and discuss how our work differs from the existing methods. 2.1 Perceptual Metric Learning While there is extensive recent research on distance metric learning, most of the algorithms are specific to class-based learning tasks such as classification [30] and clustering [35], which consider two objects similar if they belong to the same class. See Bellet et al. [3] for a comprehensive review. In contrast, our goal is to define a perceptual distance that captures the degree of similarity between any two objects irrespective of their classes. Recently, a whole new literature has emerged that emphasizes the importance of learning such continuous measures of similarity for various applications, e.g. for measuring image similarity [36], face recognition [5], concept learning [33,32] and perceptual embedding of objects [19,11,1]. The closest application to ours is perceptual embedding of objects, where the embedding function is learned so as to model the human-perceived inter-object similarity. While multidimensional scaling (MDS) techniques have been extensively applied for this [11,1,18], they are non-parametric and require numerical similarity measurement as inputs, which are hard to gather [14]. Recent works [36,21] address these limitations by developing parametric models using non-numeric relative comparisons. A relevant method is the triplet-based deep metric learning method of Kumari et al. [19]. Although our method borrows base metric learning architectures from [19],[19] doesn’t aim to make the metric learning algorithm data-efficient by developing an active data sampling technique. 2.2 Active Learning for Classification Active learning (AL) methods have been well explored for vision and learning tasks, see Settles [27] for a detailed review of active learning methods for class-based learning. Typically, the AL methods select a single instance with the maximum individual utility for annotation in each iteration. The utility of an instance is decided by different heuristics, e.g. uncertainty sampling [20], query-by-committee (QBC) [8], expected gradient length (EGL) [2], and model-output-change (MOC) [6]. The simplest and most widely applicable uncertainty sampling approach has been extended to modern deep learning frameworks and variational inference [29]. However, in all these methods, each sample’s utility is evaluated independently without considering dependence between them. A Unified Batch Selection Policy for Active Metric Learning 5 In batch-mode active learning, data items are assessed not one at a time but in batches, to reduce the number of times the model is retrained. To avoid selecting correlated batches, some recent attempts evaluate the whole batch’s utility by taking mutual information between samples into account. In contrast to our work, most of them are developed for classification tasks [2,17,26,24]. For example, Kirch et al. [17] define the utility score as the mutual information between data points in a batch and model parameters and then pick a subset with the maximum score. Pinsler et al. [24] formulate the active learning problem as a sparse subset selection problem approximating the complete data posterior of the model parameters. Both methods have a similar motivation to our work, but they are developed for the classification task, and their informativeness measures are not easy to extend to the metric learning task. Ash et al. [2] use the norm of the gradient at a sample to implicitely capture both informativeness and diversity, and select a subset of the farthest samples in the gradient space. This ensures both informativeness and diversity by a single gradient-based measure, which does not work well in the metric learning task, as shown by Kumari et al. [20]. Sener and Savarese [26] follow a similar strategy in a different feature space. Shui et al. [28] introduce a unified approach for training and batch selection process and explicitly define uncertaintydiversity trade-off by adopting Wasserstein distance. 2.3 Active Learning of Perceptual Metrics There are only a few works on active learning of a perceptual metric. Most of these, e.g. [31,9], are based on a single instance evaluation criterion. They define the utility of a single triplet and select a batch of the individually highest-utility triplets to annotate. In contrast, we define a utility score for a batch taking joint information between triplets into account. The closest work to ours is a very recent paper by Kumari et al. [20]. The algorithm involves a two-stage process. First, it selects an overcomplete set of individually highly informative samples, and then subsamples a less correlated subset, using different triplet-based decorrelation heuristics, as the current batch. This method, in essence, is still based on a single triplet selection strategy. In contrast, we present a new, rigorous approach to define utility for a batch as a whole based on joint entropy, providing a unified utility function to balance both informativeness and diversity. 3 Proposed Method In this section, we first briefly describe the perceptual metric learning setup and the underlying neural network-based learner called PerceptNet [19]. Next, we introduce our novel batch selection policy explicitly designed for triplet-based active metric learning. 3.1 Triplet-Based Active Metric Learning Let X = {xi }n1 represent a set of n objects, each described by a d-D feature vector xi . Also, let TL be a set of ordered triplets, where each triplet (xi , xj , xk ) indicates that the object xi is more similar to object xj than to xk . For brevity we denote (xi , xj , xk ) by ijk. We frame the perceptual metric learning problem as learning an embedding 6 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri ˆ φ : Rd → Rd , s.t. the L2 distance between any two objects in the embedding space dφ (x, y) = kφ(x) − φ(y)k reflects the perceptual distance between them. In recent work, φ is typically modeled with a neural network: in our experiments, we choose the existing PerceptNet model [19], where three copies of the same network, with shared weights, process three objects xi , xj and xk during training. The output is optimized P 2 2 with an exponential triplet loss L = TL e−(dφ (xi ,xk )−dφ (xi ,xj )) to maximize the distance margins (a.k.a “triplet margins”), as defined by the exponent, for training triplets. The number of possible triplets is cubic in the number of objects, so annotating a significant fraction of them is often intractable, e.g. in domains such as haptics and food tasting where annotation is especially slow. However, an effective embedding can be modeled with far fewer comparisons if triplets are sampled selectively based on how much information they would provide if annotated. This calls for active learning. The model is trained iteratively: batches of triplets informative to the current model are selected for annotation in each iteration, after which the model is retrained. However, the efficiency gain of selecting larger batches may be undone by correlation among triplets in a batch implying low overall information, a common issue in independent optimization of individual informativeness of each triplet. To mitigate this, prior works have studied batch decorrelation strategies for classification [17,2,24]. Recently, Kumari et al. [20] developed a decorrelation strategy for metric learning with separate steps for optimizing individual triplet informativeness and then batch diversity. However, as already noted, this work suffers from limitations related to their design choices. In contrast, we develop a method that jointly defines and optimizes the informativeness of an entire batch while implicitly ensuring diversity. The method is grounded in the Maximum Entropy Principle and leads to an attractive computational scheme. 3.2 Joint Entropy Measure for Batch Selection The key to a good batch mode active learning is an effective informativeness measure for a batch of triplets. For tractability, earlier work typically defines a measure adding up the individual informativeness scores of triplets. A popular score is the Shannon entropy of the prediction probability py of the current model trained on prior batches, for a triplet t taking one of two possible orderings y ∈ {ijk, ikj}, H(t) = P − y∈{ijk,ikj} py log py [31]. While often termed “uncertainty”, this is not a good predictor of actual model uncertainty due to possible bias in the current model [7]. Further, individually high-entropy triplets may also have high mutual information, hence simply adding up the scores may overestimate the actual utility of the batch. We propose a novel batch selection algorithm based on the joint entropy of an entire batch of triplets B, capturing their mutual dependence. We define the joint probability distribution of a set of unannotated triplets on some feature space such as their distance margins. This probability is defined using the distribution of likely models given prior batches, reducing any bias due to model training. Note that this is quite different from the prediction probability of a single fixed model, described above. The joint distribution over a set of triplets naturally captures the notion of interdependence among them. A Unified Batch Selection Policy for Active Metric Learning 3.3 7 Maximum-Entropy Model of the Joint Distribution Our goal is to postulate the joint probability distribution of unannotated triplets in a batch, preferably in a form that allows efficient computation of its entropy. We represent each triplet t by its distance margin ξt = d2φ (xi , xk ) − d2φ (xi , xj ). Then a batch, denoted B = {t1 , t2 , . . . , tb } is represented by the vector of distance margins given by ξ~B = [ξt1 , ξt2 , . . . , ξtb ].3 We assume there is uncertainty about these margin predictions arising from the fact that there is a distribution of plausible models given the previously annotated data. Hence, each distance margin ξti is a 1D random variable taking different values for different choices of model parameters φ. As discussed above, simply looking at the predicted ordering probabilities of individual triplets is both error-prone and fails to consider correlation between triplets. Fortunately, if the model is a neural network, it has been shown that random dropout yields a good Bayesian approximation of model uncertainty [7]. We stochastically apply the dropout K times to the model, evaluating ξ~B each time, to sample the joint margin vector distribution of the batch and to compute the corresponding b-dimensional mean and covariance matrix. We invoke the Maximum Entropy Principle [12] which maximizes the Shannon entropy subject to constraints on prescribed averages. The maximum entropy distribution, consistent with all prior constraints, ensures the largest amount of uncertainty with respect to unknown, and hence introduces no additional biases in the estimation. Empirical estimates of the entropy of the batch from samples are susceptible to noise, and lead to a hard combinatorial optimization over batches. So we constrain the mean and covariance matrix of the triplet margins to match their empirical values µ ~ B and ΣB and maximize the differenR tial entropy H(B) = − p(ξ~B ) log p(ξ~B )dξ~B subject to these constraints. This leads to a multivariate gaussian distribution N (~ µB , ΣB ) with entropy H(B) =
1 log (2πe)b det(ΣB ) 2 (1) Note that this score takes into account inter-triplet correlation, unlike measures depending only on individual marginals. The next task is to efficiently select an optimum batch of size b with maximum informativeness: B ∗ = arg maxB⊂TU ,|B|=b H(B), where TU is the set of currently unannotated triplets. 3.4 Greedy Algorithm for Batch Selection Since the maximization of the joint entropy function H(B) over subsets is computationally prohibitive, we use the fact that entropy is monotone increasing and submodular to justify a greedy policy which is provably (1 − 1e )-optimal by the results of Nemhauser et al. [22]. The greedy algorithm builds up the set B ∗ incrementally. In step k, we pick the triplet tk which has maximum conditional entropy given triplets Bk−1 selected in 3 Other triplet based representations are possible: we found the above to be a consistent and more useful feature in practice. 8 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri Algorithm 1 Greedy algorithm to maximize H(B) Input: Unlabeled triplets TU , batch size b, entropy function H : 2TU → R as in Eq. 1. Output: Batch B that is an (1 − 1e )-approximation to arg maxB⊂TU ,|B|=b H(B). 1: 2: 3: 4: 5: 6: B0 ← ∅, H(B0 ) = 0 for k=1,. . . , b do


tk ← arg maxt∈TU \Bk−1 log det ΣBk−1 ∪{t} /det ΣBk−1 Bk ← Bk−1 ∪ {tk } end for return Final subset Bk previous steps. Specifically, tk = arg = arg max t∈TU \Bk−1 H({t} | Bk−1 ) max H(Bk−1 ∪ {t}) − H(Bk−1 )
! det ΣBk−1 ∪{t}
. = arg max log t∈TU \Bk−1 det ΣBk−1 t∈TU \Bk−1 (2) This step is repeated |TU | times. The greedy policy has low complexity (quantified later) and scales well to large datasets. The overall batch selection algorithm is listed in 1. The remaining challenge is to efficiently compute the increment in the determinant of the covariance matrix in each step. We present a recursive algorithm for this, which also clarifies why the method selects a decorrelated batch. 3.5 Recursive Computation of Determinant of Covariance Matrix The covariance matrix is a Gram matrix, i.e. its (i, j)th element can be written as the dot product of the ith and j th vectors from a given family of vectors. This allows us to recursively compute its determinant and choose the recursion order according to the greedy policy for approximate optimization. Let ut denote the zero-mean vector of all sampled distance margins, [ξt (φ1 ), · · · , ξt (φK )] − [µt , · · · , µt ], for a single triplet t. The covariance matrix ΣBk−1 has the form U U T , where each column of U is us , s ∈ Bk−1 . In the k th step, a new row and column vector for a new triplet t are appended to U . Using the Gram matrix property, we have det(ΣBk−1 ∪{t} ) − det(ΣBk−1 ) = kũk k2 , where ũk is the normal from ut onto span{us | s ∈ Bk−1 }. Thus the scheme successively maximizes the squared projection error kũk k2 , over the remaining vectors {ut | t ∈ TU \ Bk−1 }. Thus we select at each step the triplet that is least correlated with the already chosen triplets. The orthogonal projections are computed using the modified Gram-Schmidt orthogonalization scheme from [10], with complexity dn2 , where d is the dimension of the ambient vector space and n the number of vectors. Since we compute the projection error for all |TU | − n ≈ |TU | remaining triplets at each step (because |TU | >> n), the overall complexity of the scheme is dn2 |TU |. A Unified Batch Selection Policy for Active Metric Learning 9 In summary, the submodularity of the joint entropy function naturally combines informativeness, diversity, and representativeness, which are precisely the desired properties for batch mode active learning. 4 Experiments We perform several experiments to answer the following questions: (1) Is our method competitive with standard baselines, including the state-of-the-art method(s), for different choices of hyperparameters, feature dimension, applications, and datasets? (2) How good is our assumption that the second-order statistics (mean and covariance) are sufficient statistics for estimating the reasonable distribution? (3) How robust is our method to labeling error? We address these questions by conducting several experiments on real-world datasets with different modalities: image, food and haptic. For each of these datasets, we select an appropriate neural network architecture – for the haptic and food datasets we ensure that these architectures exactly match those of Kumari et al. [20] so that the comparison is fair ([20] did not present any result on images, requiring us to implement their method on image databases). We test with different initial pools and varying batch sizes. We also simulated random errors in the triplet orderings to test robustness to labeling error. Datasets. We evaluate the performance of our method on five real-world datasets for which triplets defining perceptual metrics are available: Yummly food dataset [34]; TUM haptic texture dataset [30]; Abstract500 image dataset [25]; CUB-200 image dataset [32], and Scoot facial sketch dataset [5]. The Yummly-Food dataset has 72148 triplets defined over 73 food items based on taste similarity. Each food item is represented by a 6D feature vector (this is an experiment with a low feature dimension) with each component indicating different taste properties. We use 20K training and 20K test triplets sampled from the entire set of triplets. The TUM-Haptic dataset contains signals from 108 different types of surface materials. Each type of material has 32-D spectral feature vectors for 10 representative acceleration traces. The triplets are generated from a given ground-truth perceptual matrix, which has user-recorded perceived similarity responses. Like the Yummly-Food dataset, we have training and test sets of 20K triplets each. We also evaluate our method on a comparatively larger dataset (but relatively small for image data), the Abstract500 image dataset [25], which contains 500 images of 128 × 128 pixels, with pairwise perceptual similarities between them. Each image is represented by a 512-D GIST feature (an example of a relatively highdimensional feature vector) extracted using 32 Gabor filters at four scales and eight orientations [23]. We use perceptual matrix to generate 20K training and 20K test triplets. Next, we use the popular and much larger CUB-200 bird database that contains 200 bird species with roughly 30 images in each class. We choose five representative images for each class and generate its features using a pretrained ResNeXt-101-32x8d model. The network takes segmented images as input and outputs 2048-D feature vectors. The training and test sets each have 10K triplets sampled from the entire set of 93530 triplets. Finally, the results on the Scoot dataset, which is relatively small, consisting of just 1282 triplets, are presented in the supplementary material because of space constraints. 10 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri Fig. 3: Performance of different active learning methods on two image datasets CUB200 and Abstract500 for increasing batch sizes (400/600/800 or 200/500/800, from left to right). Here accuracy means what fraction of test triplets have been detected with the correct ordering. To avoid clutter, standard deviations are shown only for the CUB-200 dataset and the rest are shown in the supplementary material. Baselines. We compare our method with five baselines, including the state-of-the-art method: (1) US-hDisti: A batch of individually high-entropy triplets is pruned subjected to different (denoted by hDisti) decorrelation measures to select a diverse batch of informative triplets [20]. It is the current state-of-the-art for batch mode active metric learning, and outperforms other alternatives like BADGE [2] (adapted to metric learning). We pick hDisti to be the highest-performing variant in each individual experiment. (2) Variance: Triplets with the highest individual distance-margin variance across a collection of models generated using dropout [13]. This method simulates the effect of replacing the joint entropy of a batch with the sum of individual entropies of triplets in the batch. (3) Random: A passive learning strategy that uniformly samples each batch of triplets at random. Though naı̈ve, this choice often results in reasonably good accuracy. (4) US: Uncertainty method, which picks the top b triplets with highest uncertainty in predicted triplet ordering (i.e. the model’s (lack of) ordering confidence), without taking correlation among them into account [31]. (5) BADGE: A diverse set of triplets with maximum loss gradients for the most probable label, selected using k-means ([2] adapted to the triplet scenario). Active Learning Setup. For the CUB-200 dataset, we begin each experiment with an initial pool of 1000 annotated triplets, and for the other three datasets with 600 annotated triplets, and pretrain the model φ0 , which is used as a common starting point for all compared methods. In each active learning iteration, we select the best fixedsize batch of unannotated triplets, using the chosen batch selection method, and acquire their orderings. To make convergence faster, we update the current model φi to obtain φi+1 using the available additional annotated triplets instead of training ab initio. The performance of the learned model is evaluated by its triplet generalization accuracy, A Unified Batch Selection Policy for Active Metric Learning 11 Fig. 4: Performance of different active learning methods on Yummly-Food and TUMHaptic texture datasets for varying batch sizes (500/800/1000 or 400/600/800). which denotes the fraction of triplets whose ordering is correctly predicted [19]. Each experiment is repeated with five random train/test splits, and the average performance along with the standard deviation is reported. (For most plots, the standard deviation is shown in supplementary material, for clarity.) Implementation Details. The architecture and training hyperparameters used for different datasets are as follows: Yummly-Food: 3 fully-connected (FC) layers with 6, 12 and 12 neurons; TUM-Haptic: 4 FC layers with 32, 32, 64 and 32 neurons; Abstract500Image: 6 FC layers with 512, 256, 128, 64, 32 and 16 neurons; CUB-200: 3 FC layers with 2048, 512 and 32 neurons. Each layer is followed by a dropout layer with a dropout probability of 0.02. The Adam optimizer [16] is used for training all models with a learning rate of 10−4 for Yummly-Food, TUM-Haptic, and Abstract500-Image dataset, and 10−5 for CUB-200 dataset. The model is trained with an SGD batch size of 500 for 1000 epochs for all four datasets. Active Learning Performance. The performance of our method against the baselines described earlier is plotted in Figure 3 for the image datasets CUB-200 and Abstract500Image, shown as a function of the number of active learning iterations. In Figure 4, we compare the performances of all methods for the data from other modalities, i.e., haptic and food. We observe that our method is consistently better than the state-of-the-art US-hDisti method (for clarity, we only show the specific variant offering the best performance in each experiment). Our method reaches higher accuracies quicker and also tends to converge to a higher final accuracy on both Yummly-Food and TUM-Haptic datasets. For the large CUB-200 image dataset, our method is neck-and-neck with the state-of-the-art for the first few iterations and then rapidly overtakes it, widening the gap with additional iterations. For the smaller Abstract500 image dataset, the improvements are more prominent with larger batch sizes, reflecting the focus of our work on batch-mode learning. Additionally, for the CUB-200 dataset, we plot the standard devi- 12 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri Query Proposed method Random sampling Query Proposed method Random sampling Fig. 5: Top-4 retrieved images in the order of increasing perceptual distance, left to right, using our method and random sampling (randomly-selected batches are annotated for training) on different modalities datasets. On both datasets, each model is trained for twelve training rounds, constituting 18% of training triplets. Images different from the query class are bounded by a red box, substantiating that two images from different classes can be perceptually more similar than two from the same class. The triplet order accuracy, defined here as the number of test triplets whose order is preserved by the ranked list of retrieved images, for our method vs random sampling after the 12th round 12 12 12 of training is MOurs = 96.7%, MRandom = 92% for image dataset and MOurs = 72.9%, 12 MRandom = 69.3% for food dataset. More results with different queries and learned metrics are shown in the supplementary material. ation in the same plot as the shaded region (of the same color) around the performance curves for different methods (standard deviations on other datasets are shown in supplementary). Even though the figure looks a little cluttered, one can see that the standard deviation for the proposed method is better than that of the next-best method, signifying a more consistent performance. This substantiates our claim that joint entropy is a better batch score than an ad-hoc combination of independent informativeness and diversity heuristics. Further, our method does not require the user to select a suitable decorrelation heuristic to manually fine-tune the performance. We also outperform the other two baselines: Random and Variance. It is particularly informative to see the generally poor performance of Variance (lower than the Random). Because of high correlations among informative triplets, individually selecting the most informative triplets does not learn the entire metric space as well as just picking triplets at random. In contrast, our method as well as that of Kumari et al. [20] both incorporate batch decorrelation and outperform random sampling. This shows the critical importance of batch diversity in an active learning strategy. Next, we evaluate the effectiveness of our method for an object retrieval task. Specifically, we compare our method with the random sampling baseline at different training rounds. We show the retrieval results on two different modalities, food and image. We split the Yummly-Food dataset into 40000 training and 32148 test triplets, and A Unified Batch Selection Policy for Active Metric Learning 13 Fig. 6: Ablation study to analyze the robustness of our method to different values of K (# of prior models sampled by dropout) and p (dropout probability) on different datasets. the CUB-200 dataset into 40000 training and 33000 test triplets. On both datasets, we perform active learning with a batch size of 600 and an initial pool of 500 triplets. For a given query image, the top four instances from the retrieval set are shown (ranked from most similar to least similar) in Figure 5. As we can see, retrieval results of our method resemble the query in taste or visual appearance better than the random sampling. Please see the supplementary material for further results from this experiment. Ablation Study. In order to get an estimate of the covariance matrix, we perform random dropouts in the neural network K times. Naturally, as K increases, one gets a better estimate of the covariance matrix. However, this may increase the computation time. We perform an ablation study to see how this hyperparameters (i.e., variation in K and dropout probability p) affect the triplet order accuracy, and the results are shown in Figure 6 for three different modalities: image, haptic and food. It can be seen from the plots that a moderate value of about K = 70 or 100 is good enough as the performance is not significantly dependent on the choice of K. We also observe that the performance is robust to variation in dropout probability; however, there is a significant variation for the Yummly-Food dataset, with optimal p = 0.02. Runtime Analysis. We also compare the computational requirement of the proposed method with that of Kumari et al. [20]. The key computational step in [20] involves searching for the subset of maximally apart (in the feature space) triplet at each training round, apart from the computation of the gradients. They also use a greedy search technique for subset selection. For the proposed method, the subset selection process is efficient, but the computation of determinant of covariance matrix at each iteration does consume a good amount of time. Overall, both the methods were found to consume a nearly equal amount of computation time when the Gram-Schmidt orthogonalization is used. For instance runtimes (in secs) of different batch selection policies to select a 500-triplet batch from Yummly-food are: US: 0.109, Variance: 0.083, BADGE: 60.104, 14 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri US-hDisti: 8.110, Ours: 7.803. While the computation complexity varies with the feature dimension and model size, the relative performance remains similar. Robustness to Labeling Error. To evaluate the robustness of our method against labeling error, we corrupt 10% and 30% of the ground-truth training triplets in the food and image datasets by flipping their orders. Figure 7 shows how the noisy training set affects the performance of our Fig. 7: Performance of our method vs random sammethod vs the random sampling pling in the presence of labeling error. baseline. For the food data (top), with a relatively low 10% labeling error, our method takes slightly more iterations to gain accuracy, but eventually converges to a comparably high accuracy as the noise-free case, while random batch selection fails to achieve the same performance even with clean data. As the percentage of noisy triplets increases, the performance of both methods degrade, showing vulnerability to large scale labeling error. In the absence of abnormally high levels of outliers, our method shows robust performance. For the more complex image dataset (bottom), labeling error has a stronger negative effect (the first selected batch actually decreases overall accuracy), but at each noise level our method still outperforms the baseline. Comparison of Data Distribution to Theoretical Distribution. We study the validity of the Gaussian embedding, though it already has justification as “worst-case analysis” due to the Maximum Entropy Principle. A standard test is the quantile-quantile (QQ) plot [4], which indicates how close the empirical distribution is to the theoretical distribution. For ease of visualization, we show the QQ plot and histogram for a single randomly-selected unlabeled triplet, for a particular model trained on the initial triplet pool in each dataset. (We cannot visualize a full multivariate QQ plot over all possible batches.) In the QQ plot, the x-axis denotes the theoretical quantiles, which in our case is a Gaussian distribution with the empirical mean and variance, and the observed ordered distance margins are on the y-axis. The goodness of fit is indicated by the alignment of points with the straight line having a unit slope. As shown in Figure 8, in all four datasets, the plotted curve closely approximates the corresponding straight lines shown in red. Our approximation is further validated in the histogram, where our data distribution shows a reasonable fit with the theoretical distribution (shown in green) for the most part, except that the actual distribution is a little more peaked. 5 Conclusion, Limitations, and Future Work We have introduced a novel approach for batch-mode active metric learning based on maximizing the joint entropy of a batch. We found that a batch of individually informative triplets does not form an optimal subset, even if decorrelation heuristics are applied to reduce their correlation. Instead of defining separate measures for informativeness A Unified Batch Selection Policy for Active Metric Learning 15 Fig. 8: QQ plot and histogram for all four datasets to demonstrate how closely the actual distribution follows the theoretical distribution. and diversity, our method defines the joint entropy of a batch of triplets as a unified measure that jointly optimizes both. The overall method involves no heuristic parameter selection and has no control parameter to tweak, other than the number of dropout samples and dropout probability, once the network architecture is chosen. While our method shows promising results, it does have a few limitations. First, approximating the joint distribution of data using the Maximum Entropy Principle gives the most general distribution for a given prior, which in the case of second-order statistics as constraints is a Gaussian. However, in some cases, where the actual distribution may be quite non-Gaussian, the joint entropy measure defined with the second-order statistics may misguide the batch selection policy. One important direction for future work is extending our framework beyond second-order statistics to learn the joint distribution of data closer to empirical distribution. Another important extension would be to modify our framework to dynamically learn the optimal batch size and batch selection policy, which we believe would further improve the performance and generalize well to diverse inputs and applications. References 1. Agarwal, S., Wills, J., Cayton, L., Lanckriet, G., Kriegman, D., Belongie, S.: Generalized non-metric multidimensional scaling. In: AIS (2007) 4 2. Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. In: ICLR (2020) 4, 5, 6, 10 3. Bellet, A., Habrard, A., Sebban, M.: A survey on metric learning for feature vectors and structured data. arXiv preprint arXiv:1306.6709 (2013) 4 4. Ben, M.G., Yohai, V.J.: Quantile–quantile plot for deviance residuals in the generalized linear model. Journal of Computational and Graphical Statistics (2004) 14 5. Fan, D.P., Zhang, S., Wu, Y.H., Liu, Y., Cheng, M.M., Ren, B., Rosin, P.L., Ji, R.: Scoot: A perceptual metric for facial sketches. In: ICCV (2019) 4, 9 6. Freytag, A., Rodner, E., Denzler, J.: Selecting influential examples: Active learning with expected model output changes. In: ECCV (2014) 4 7. Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: ICML (2016) 3, 6, 7 8. Gilad-Bachrach, R., Navot, A., Tishby, N.: Query by committee made real. In: NeurIPS (2006) 4 16 Priyadarshini K, Siddhartha Chaudhuri, Vivek Borkar, and Subhasis Chaudhuri 9. Heim, E., Berger, M., Seversky, L., Hauskrecht, M.: Active perceptual similarity modeling with auxiliary information. In: AAAI (2015) 3, 5 10. Hoffmann, W.: Iterative algorithms for gram-schmidt orthogonalization. Computing (1989) 8 11. Jamieson, K.G., Nowak, R.D.: Low-dimensional embedding using adaptively selected ordinal data. In: Allerton (2011) 4 12. Jaynes, E.T.: Information theory and statistical mechanics. Physical Review (1957) 7 13. Kendall, A., Badrinarayanan, V., Cipolla, R.: Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv preprint arXiv:1511.02680 (2015) 10 14. Kendall, M.G.: Rank correlation methods. Griffin (1948) 2, 4 15. Kim, S., Seo, M., Laptev, I., Cho, M., Kwak, S.: Deep metric learning beyond binary supervision. In: CVPR (2019) 2 16. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015) 11 17. Kirsch, A., van Amersfoort, J., Gal, Y.: BatchBALD: Efficient and diverse batch acquisition for deep Bayesian active learning. In: NeurIPS (2019) 5, 6 18. Kruskal, J.B., Wish, M.: Multidimensional scaling. Elsevier (1978) 2, 4 19. Kumari, P., Chaudhuri, S., Chaudhuri, S.: PerceptNet: Learning perceptual similarity of haptic textures in presence of unorderable triplets. In: WHC (2019) 2, 4, 5, 6, 11 20. Kumari, P., Goru, R., Chaudhuri, S., Chaudhuri, S.: Batch decorrelation for active metric learning. In: IJCAI-PRICAI (2020) 1, 3, 4, 5, 6, 9, 10, 12, 13 21. McFee, B., Lanckriet, G.: Learning multi-modal similarity. JMLR (2011) 4 22. Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximizing submodular set functions – I. Mathematical programming (1978) 3, 7 23. Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. IJCV (2001) 9 24. Pinsler, R., Gordon, J., Nalisnick, E., Hernández-Lobato, J.M.: Bayesian batch active learning as sparse subset approximation. In: NeurIPS (2019) 5, 6 25. Robb, D.A., Padilla, S., Kalkreuter, B., J.Chantler, M.: Crowdsourced feedback with imageryrather than text: Would designers use it? In: SIGCHI (2015) 9 26. Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core-set approach. In: ICLR (2018) 5 27. Settles, B.: Active learning. SLAIML (2012) 4 28. Shui, C., Zhou, F., Gagné, C., Wang, B.: Deep active learning: Unified and principled method for query and training. In: AISTATS (2020) 5 29. Sinha, S., Ebrahimi, S., Darrell, T.: Variational adversarial active learning. In: ICCV (2019) 4 30. Strese, M., Boeck, Y., Steinbach, E.: Content-based surface material retrieval. In: WHC (2017) 1, 4, 9 31. Tamuz, O., Liu, C., Belongie, S., Shamir, O., Kalai, A.T.: Adaptively learning the crowd kernel. In: ICML (2011) 3, 5, 6, 10 32. Wah, C., Maji, S., Belongie, S.: Learning localized perceptual similarity metrics for interactive categorization. In: WACV (2015) 4, 9 33. Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., Wu, Y.: Learning fine-grained image similarity with deep ranking. In: CVPR (2014) 1, 4 34. Wilber, M.J., Kwak, I.S., Belongie, S.J.: Cost-effective hits for relative similarity comparisons. In: AAAI (2014) 9 35. Xing, E.P., Jordan, M.I., Russell, S.J., Ng, A.Y.: Distance metric learning with application to clustering with side-information. In: NeurIPS (2003) 1, 4 36. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018) 2, 4 A Unified Batch Selection Policy for Active Metric Learning Supplementary Material In this supplementary material, we provide (a) plots with standard deviations for three datasets where only the mean accuracies were provided in the main paper, for clarity; (b) performance evaluation of different active learning methods on one additional dataset and (c) qualitative and quantitative results on a retrieval task with metrics trained with different methods. 1 Standard Deviation Plots Fig. 1: Mean accuracy and standard deviation plots (computed over five random train/test splits) of different active learning methods for Abstract500-Image, YummlyFood and TUM-Haptic datasets with increasing batch sizes. 18 Supplementary Material In Figure 1, we augment the plots shown in the main paper with standard error bands across the five experimental runs for each method+hyperparameter combination for three real-world datasets: Abstract500-Image, Yummly-Food, and TUM-Haptic. (Bands for CUB-200 are directly shown in the main paper.) The standard error of our method is small for all datasets except the Yummly-Food dataset, where all methods have high variance. While our method always exceeds or matches the best alternative, the performance gain is observed to be higher for a larger batch size. 2 Performance of Active Learning Methods on Scoot Facial Sketch Dataset The Scoot facial sketch dataset is relatively small, consisting of just 1282 triplets, where each triplet represents similarity ordering between three sketched faces of a person. The facial sketch is represented by a 512-D GIST feature extracted using 32 Gabor filters at four scales and eight orientations. The training and test set contains 800 and 200 triplets, respectively, sampled from the entire triplet set. The architecture and training hyperparameters used for Scoot facial sketch dataset are: 6 FC layers with 512, 256, 128, 64, 32 and 16 neurons. Each layer is followed by a dropout layer with a dropout probability of 0.02. The Adam optimizer is used for training the model with a learning rate of 10−6 . The model is trained with an SGD batch size of 500 for 1000 epochs. As can be seen in Figure 2, in the initial training rounds with only a few annotated triplets, our method performs marginally better than other baselines, but the accuracy margin improves as the number of annotated triplets increases. Fig. 2: Performance of different active learning methods on the Scoot dataset. Test accuracy indicates the fraction of test triplets correctly ordered by the learned perceptual metric. A Unified Batch Selection Policy for Active Metric Learning 3 19 Image Retrieval Task In this section, we further evaluate the effectiveness of our method for an image retrieval task. We compare our method with the random sampling baseline at different training rounds on the CUB-200 and Yummly-Food datasets. 3.1 Retrieval Results on CUB-200 Dataset The CUB-200 dataset consists of 200 classes with roughly 30 images in each class. Triplets are defined based on visual (perceptual) similarity of classes. We split the dataset into 40000 training and 33000 test triplets, and performed active learning with a batch size of 600 and an initial pool of 500 triplets. For a given query image, the top four instances from the retrieval set are shown (ranked from most similar to least similar) in Figures 3 and 4, after two different training rounds. 3.2 Retrieval Results on Yummly-Food Dataset The Yummly-Food dataset consists of images of 73 food items and 72148 triplets defined on the basis of similarity in taste. We split the dataset into 40000 training and 32148 test triplets, and performed active learning with a batch size of 600 and an initial pool of 500 triplets. For a given query image, the top nine instances from the retrieval set are shown (ranked from most similar to least similar) in Figures 5 (9 training rounds) and 6 (20 training rounds). 20 Supplementary Material Fig. 3: Performance comparison between our method and random sampling for the image retrieval task after two different training rounds. The leftmost column presents a query, and the remaining columns present the first four retrieved images in the order of increasing perceptual distance, left to right. Images with red squares belong to classes different from the query class. Our results indicate that two images from different classes can be visually more similar than two from the same class, highlighting the distinction between perceptual and class-based metrics. The triplet ordering accuracy k for MOurs/Random (model trained on triplets selected by our method vs random samth 9 9 12 pling resp. at k training round): MOurs = 95.3%, MRandom = 90.3%, MOurs = 96.7%, 12 MRandom = 92%. A Unified Batch Selection Policy for Active Metric Learning 21 Fig. 4: Similar results as Figure 3, shown for a different set of query images in the leftmost column. 22 Supplementary Material (a) Retrieval results of our method using annotations of 18% of training triplets. (b) Retrieval results of the random sampling baseline using annotations of 18% of training triplets. Fig. 5: Top-9 food dishes, retrieved according to taste similarity, by our method vs random sampling after twelve training rounds, for query images in the leftmost columns. k The triplet ordering accuracy for MOurs/Random (model trained on triplets selected 12 by our method vs random sampling resp. at k th training round): MOurs = 72.9%, 12 MRandom = 69.3%. Note that both our method and random sampling solicit annotations of 18% of the training triplets; however, our method’s retrieval results better resemble the query in taste, than those from random sampling. A Unified Batch Selection Policy for Active Metric Learning 23 (a) Retrieval results of our method using annotations of 30% of training triplets. (b) Retrieval results of the random sampling baseline using annotations of 30% of training triplets. Fig. 6: Top-9 retrieved food dishes by our method vs random sampling after twenty training rounds, for query images in the leftmost columns. The triplet ordering accuracy: 20 20 MOurs = 73.6%, MRandom = 69.7%.