CL²R: Compatible Lifelong Learning Representations

The universe is dynamic, and the emergence of novel data and new knowledge is unavoidable. The unique ability of natural intelligence to lifelong learning is highly dependent on memory and knowledge representation [18]. Through memory and knowledge representation, natural intelligent systems continually search, recognize, and learn new objects in an open universe after exposure to one or a few samples. Memory is substantially a cognitive function that encodes, stores, and retrieves knowledge. Artificial representations learned by Deep Convolutional Neural Network (DCNN) models [3, 61, 63, 64, 76] stored in a memory bank (i.e., the gallery-set) have been shown to be quite effective in searching and recognizing objects in an open-set/open-world learning context. Successful examples are face recognition [10, 14, 59], person re-identification [78, 79, 80], and image retrieval [19, 65, 73].

These approaches rely on learning feature representations from static datasets in which all images are accessible at training time. However, dynamic assimilation of new data for lifelong learning suffers from catastrophic forgetting: the tendency of neural networks to abruptly forget previously learned information [37, 52].

In the case of visual search, even avoiding catastrophic forgetting by repeatedly training DCNN models on both old and new data, the feature representation still irreversibly changes [31]. Thus, to benefit from the newly learned model, features stored in the gallery must be reprocessed and the “old” features replaced with the “new” ones. Reprocessing not only requires the storage of the original images (a noticeable leap from natural intelligence) but also their authorization to access them [66]. More importantly, extracting new features at each update of the model is computationally expensive or infeasible in the case of large gallery-sets. The speed at which the representation is updated to benefit from the newly learned data may impose time constraints on the re-indexing process. This may occur from timescales on the order of weeks/months as in retrieval systems or social networks [62], to within seconds as in autonomous robotics or real-time surveillance [43, 48]. Recently, in the work of Shen et al. [62], a novel training procedure was proposed to avoid re-indexing the gallery-set. The representation obtained in this manner is said to be compatible, as the features before and after the learning upgrade can be directly compared. Training takes advantage of all data from previous tasks (i.e., no lifelong learning), guaranteeing the absence of catastrophic forgetting. The advantage of considering compatible representation learning within the lifelong learning paradigm, as in this work, is that compatible representation allows visual search systems not only to distribute the computation over time but also to avoid or possibly limit the storage of images on private servers for gallery data. This can have important implications for the societal debate related to privacy, ethical, and sustainable issues (e.g., carbon footprint) of modern AI systems [11, 49, 60, 66].

We identify stationarity as the key requirement for feature representation to be compatible during lifelong learning. Stationary features have been shown to be biologically plausible in many studies of working memory in the prefrontal cortex of macaques [33, 39, 40]. Some works [39, 40] decoded the information from the neural activity of the working memory using a classifier with a single fixed set of weights. They noted that a non-stationary feature representation seems to be biologically problematic since it would imply that the synaptic weights would have to change continuously for the information to be continuously available in memory.

Inspired by this, in this article, we formalize the problem of Compatible Lifelong Learning Representations ( \(\textbf {CL}^{\bf 2}{\bf R}\) ) in relation to the relevant areas of compatible learning and lifelong (continual) learning. We call any training procedure that aims to obtain compatible features and minimize catastrophic forgetting as CL²R training, and we propose (1) a novel set of metrics to properly evaluate CL²R training procedures, (2) a training procedure based on rehearsal [52, 54], and feature stationarity [46, 47] to jointly address catastrophic forgetting and feature compatibility. Figure 1 provides an overview of the problem and the training procedure. Specifically, our CL²R training procedure is achieved by encouraging global and local stationarity to the learned features.

The rest of the article is organized as follows. In Section 2, we discuss related work, and in Section 3, we highlight our contributions. Section 4 presents the formulation of CL²R, Section 5 proposes new metrics to evaluate compatibility, and Section 6 describes a new training procedure. In Section 7, we compare our results with adapted state-of-the-art methods. Section 8 presents the ablation study. We conclude in Section 9.

Compatible learning. The work proposed by Shen et al. [62], called Backward-Compatible Training (BCT), first formalizes the problem of learning compatible representation to avoid re-indexing. The method takes advantage of an influence loss that encourages the feature representation toward one that can be used by the old classifier. The old classifier is fixed while learning with the novel data (i.e., its parameters are no longer updated by back-propagation) and cooperates with the new representation model. Cooperation is achieved by aligning the prototypes of the new classifier with the prototypes of the old fixed one. The underlying assumption is that the upgraded feature representation follows the representation learned by the old classifier. BCT has been evaluated in scenarios without the effects of catastrophic forgetting by repeatedly training DCNN models on both old and new images (i.e., jointly re-training from scratch at each upgrade). To compare with this learning strategy in a lifelong learning scenario instead of starting from scratch every time, we have added to BCT the capability of learning by fine-tuning the previously learned model according to a memory-based rehearsal strategy [52, 54].

Compatibility under catastrophic forgetting has been implicitly studied in the work of Iscen et al. [25] (FAN), in which authors presented a method for storing features instead of images in Class-incremental Learning (CiL). They introduce a feature adaptation function to update the preserved features as the network learns novel classes. We compared to this method by storing the updated-preserved features obtained at each task. Although designed to improve classification accuracy, the work can be considered close to a lifelong learning approach with compatible representation in which the feature adaptation function they defined addresses implicitly the problem of feature compatibility as in other works [6, 23, 38, 68]. Differently from BCT, these methods do not completely prevents the cost of re-indexing since the learned mappings require evaluation every time the dataset is upgraded and are therefore they are not suited to lifelong learning and/or large gallery-set. For example, the ResNet-101 architecture is one order slower than the mapping proposed in the work of Chen et al. [6]; therefore, when the size of the gallery increases by an order of magnitude, it is equivalent to re-indexing the images. The method described in the work of Ramanujan et al. [51], in addition to the current feature model, trains from the same data an auxiliary model in a different way (i.e., using self-supervised learning). The auxiliary feature model will then be used with future learned models to learn a mapping model to obtain compatible representations as in other works [25, 38, 68]. The underlying assumption is that as the auxiliary feature model is trained with a different strategy, it encodes different knowledge that may facilitate learning the mapping between the representation spaces.

Compatibility of the representation in a more general sense has been considered in the work of Li et al. [31] and Wang et al. [70], where similarity between features extracted from identical architectures and trained from different initialization has been extensively evaluated. The work of Budnik and Avrithis [5] avoids re-indexing the gallery, although the new model used for queries is not trained on more data. Their work is motivated by the scenario where the gallery is indexed by a large model and the queries are captured from mobile devices in which the use of small models is the only viable solution.

Lifelong learning. Lifelong learning or continual learning studies the problem of learning from a non-i.i.d. stream of data with the goal of assimilating new knowledge preventing catastrophic forgetting [9, 37]. Methods for preventing catastrophic forgetting have been explored primarily in the classification task, where catastrophic forgetting often manifests itself as a significant drop in classification accuracy [2, 13, 35, 41, 67]. The key aspects that distinguish lifelong feature learning for visual search from classification are the following: (i) categorical data often have coarser granularity than visual search data, (ii) evaluation metrics do not involve classification accuracy, and (iii) class labels are not required to be explicitly learned. These differences may suggest that these two catastrophic forgetting occurrences are of different origins. In this context, recent works have discussed the importance of the specific task in assessing catastrophic forgetting of learned representations [1, 7, 8, 12, 47, 50]. Among others, empirical evidence presented in the work of Davari and Belilovsky [12] suggests that feature forgetting is not as catastrophic as classification forgetting and that many approaches that address the problem of catastrophic forgetting do not improve feature forgetting in terms of the usefulness of the representation. We argue that such evidence is relevant in visual search and that it can be exploited with techniques that further encourage learning compatible feature representation. According to this, we consider CiL as the basic building block for the general purpose of learning feature representation incrementally.

In this article, the focus is on CiL methods based on Knowledge Distillation (KD) [21] and rehearsal [55], which are known to be versatile, effective, and widely applicable to reduce catastrophic forgetting. We leverage the classification task in CiL as a surrogate task to learn feature representation as typically performed in face/body identification and retrieval [14, 65, 79]. The work of Li and Hoiem [32] first introduces KD in lifelong learning as an effective way to preserve the knowledge previously acquired from old tasks. In iCaRL [53], KD is combined with rehearsal to reserve samples of exemplars stored in an episodic memory for classes already seen. The BiC work, proposed by Wu et al. [71], extends the work of Rebuffi et al. [53] by developing a bias correction layer to recalibrate the output probabilities learning an additional linear layer on a small set of data. Along a similar vein, in the work of Zhao et al. [77], the bias correction is performed by aligning the norms of the weight vectors of the classifier for new classes to those for old classes without using additional model parameters or reserved data. The work of Romero et al. [56] introduces Feature Distillation (FD), a distillation loss evaluated on the feature vectors instead on the classifier outputs. FD has recently been successfully applied by Hou et al. [22] (LUCIR) and Douillard et al. [16] to reduce catastrophic forgetting. Differently from LUCIR, PODNet uses a spatial-based distillation loss to constrain the statistics of intermediate features after each residual block. Similar to LUCIR, PODNetm and many others works on continual/lifelong learning in the literature, our problem formulation takes advantage of the general concept of KD. Differently from these works, our approach is novel in that it considers FD for the dual purpose of learning feature compatibility and mitigating feature forgetting. The work of Iscen et al. [25] (FAN), also discussed in the previous paragraph, combines strategies from other works [22, 32, 53] to learn and preserve previous features. Although the work does not consider the compatibility problem, it is the closest work to our approach. Recently, Yan et al. [72] (DER) showed an interesting performance improvement in CiL by freezing the previously learned representation and expanding its dimension from a new learnable feature extractor. Despite the clear improvements in classification performance, this has no trivial exploitation in compatible training, as the varying dimensions across tasks do not allow direct application of nearest-neighbor search between models. Features with different dimensions typically require to be projected into a common single space to allow nearest-neighbor to be applied. The FOSTER method [69] improves upon DER by addressing this specific problem by transforming the growing dimension of the feature representation with a trainable linear layer that maps the growing feature vector into a fixed dimension. More in general, CiL methods addressing catastrophic forgetting are in a certain sense related to compatible representation, since forgetting is the change in the feature representation of classifiers that will be learned in the future. We evaluate these methods as baselines to quantify the level of lifelong-compatible representation they intrinsically may have.

In a CL²R setting, a sequence of representation models, \(\lbrace \phi _t \rbrace _{t=1}^{T}\) , is learned incrementally with a sequence of T tasks, \(\lbrace (\mathcal {D}_t, K_t) \rbrace _{t=1}^T\) , where \(\mathcal {D}_t\) are the images of the t-th task represented by \(K_t\) different classes. Specifically, each task is disjoint from the others: \(K_k \cap K_t= \emptyset\) with \(t \ne k\) . The learned representation model \(\phi _t\) is used to transform the query images into feature vectors that are used to retrieve the images most similar to a set of given gallery images transformed with a previous model \(\phi _k\) . Specifically, we indicate with the couple \(\mathcal {G}=(I_\mathcal {G},F_\mathcal {G})\) the gallery-set, where \({I}_\mathcal {G}=\lbrace \mathbf {x}_i\rbrace _{i=1}^N\) is the image collection from which the features \(F_\mathcal {G}=\lbrace \mathbf {f}_i \rbrace _{i=1}^N\) are extracted, and N is the number of elements of the two sets. Without loss of generality, we assume that the features in \(F_\mathcal {G}\) are extracted using the representation model \(\phi _{ k}:{\mathbb {R}}^D \rightarrow {\mathbb {R}}^d\) that transforms an image \(\mathbf {x} \in {\mathbb {R}}^D\) into a feature vector \(\mathbf {f} \in {\mathbb {R}}^d\) , where d and D are the dimensionality of the feature and the image space, respectively. Analogously, we will refer to \(\mathcal {Q}=(I_\mathcal {Q},F_\mathcal {Q})\) as the query-set, where \(I_\mathcal {Q}\) and \(F_\mathcal {Q}\) are the corresponding image-set and the feature-set, respectively. As the t-th task becomes available, the model \(\phi _{t}\) is incrementally learned from the previous one along with the new task data \(\mathcal {D}_t\) . Our goal is to design a training procedure to learn the model \(\phi _{t}\) so that any query image transformed with it can be used to perform visual search through some distance \({\rm dist}:{\mathbb {R}}^d \times {\mathbb {R}}^d \rightarrow \mathbb {R}_+\) to identify the closest features \({F}_\mathcal {G}\) to the query features \({F}_\mathcal {Q}\) without forgetting the previous representation and without computing \(F_\mathcal {G}=\lbrace \mathbf {f} \in \mathbb {R}^d \, | \, \mathbf {f} = \phi _{t}(\mathbf {x}) \, \forall \mathbf {x} \in I_\mathcal {G}\rbrace\) (i.e., re-indexing). If this holds, then the resulting representation \(\phi _{t}\) is said to be lifelong compatible with \(\phi _{k}\) .

The main challenge of the CL²R problem is to jointly alleviate catastrophic forgetting and learn a compatible representation between the previously learned models. In Figure 1, we illustrate the complete CL²R training example using rehearsal to alleviate the effects of catastrophic forgetting.

A representation model \(\phi _{\rm new}\) upgraded with new data is said to be compatible with an old representation model \(\phi _{\rm old}\) when it holds [62]:

Equation (1) represents the Empirical Compatibility Criterion (ECC), where \({M}\) is an evaluation metric specific to the given visual search problem. Notable examples of the metric M can be found in face verification accuracy [24, 30], face verification/identification accuracy in terms of true acceptance rate and false acceptance rate (TAR \(@\) FAR) [27], and person re-identification mean average precision (mAP) [74]. The intuition of these metrics is based on the observation that they can be instantiated with two different representation models \(\phi _{\rm new}\) and \(\phi _{\rm old}\) when considering the query-gallery pair. The specific notation \({M} (\phi _{\rm new}^{\mathcal {Q}}, \phi _{\rm old}^{\mathcal {G}})\) defines the cross-test between the new and the old model, and it represents the case in which \(\phi _{\rm new}\) is used to extract the features of the query-set, \(F_\mathcal {Q}\) , whereas \(\phi _{\rm old}\) is used to extract the gallery-set ones, \(F_\mathcal {G}\) . \({M} (\phi _{\rm old}^{\mathcal {Q}}, \phi _{\rm old}^\mathcal {G})\) is the self-test, and it represents the case in which both query and gallery features are extracted with \(\phi _{\rm old}\) . When the model is trained incrementally on T tasks, Equation (1) is evaluated according to the multi-model ECC introduced by Biondi et al. [4]:where \(t, k \in \lbrace 1,2,\ldots ,T\rbrace\) refer to two different tasks such that task k is processed by the model before task t. The model \(\phi _t\) is compatible with the model \(\phi _k\) , when the cross-test \(M (\phi _t^\mathcal {Q}, \phi _k^\mathcal {G})\) between \(\phi _t\) and \(\phi _k\) is greater than the self-test \(M (\phi _k^\mathcal {Q}, \phi _k^\mathcal {G})\) of the model \(\phi _k\) . The underlying intuition is that if the performance of matching the gallery feature vectors extracted with the old model with the query feature vectors extracted with the new model (i.e., cross-test) is better than the performance of matching the gallery feature vectors with the query feature vectors both extracted with the old model (i.e., self-test), then the system is learning compatible representations. In other words, learning from the new task data improves the representation without breaking the compatibility with the previously learned model. Based on Equation (2), the compatibility matrix C is defined as follows:where the element in the row t and the column k of the compatibility matrix denotes the evaluation metric M of the model t to the model k. This definition combines the basic intuition of the classification accuracy matrix R defined elsewhere [15, 34], used to evaluate the CiL problem, with the two specific aspects that distinguish the \(\text{CL}^2\text{R}\) learning setting from the CiL one. Namely, (i) in CiL at each task, the train and test data are sampled from the same distribution, whereas in \(\text{CL}^2\text{R,}\) the test-set classes are sampled from an unknown distribution (i.e., \(\text{CL}^2\text{R}\) addresses the open-set recognition problem); (ii) in CiL, the test-set is dynamic (i.e., it grows including images from the task distributions), whereas in \(\text{CL}^2\text{R,}\) it is assumed static for the purpose of a reliable evaluation [62]. In the \(\text{CL}^2\text{R}\) setting, a dynamic test-set, as used in CiL, is of difficult definition, as there are infinite ways to make the gallery dynamic and each of them may change unexpectedly the performance of the evaluation. We follow Shen et al. [62] and perform the evaluation assuming a static test-set (i.e., a static query-gallery pair). According to this, we set the elements of the matrix C with \(t\lt k\) to zero to indicate the impossibility of a reliable evaluation of a growing test-set that should be sampled from an unknown changing distribution. For the remaining elements, the cross-test values are the elements of the matrix with \(t \gt k\) , whereas the self-test values are those of the main diagonal (i.e., when \(t = k\) ). Given a compatibility matrix C, the average compatibility (AC) is defined as follows:where \({1\!\!1}(\cdot)\) denotes the indicator function. AC summarizes the compatibility matrix values in a single number that quantifies the number of times that compatibility is verified against all possible \(\frac{T(T-1)}{2}\) occurrences.

To achieve compatibility, we encourage global and local stationarity to the feature representation.

Global stationarity is encouraged according to the approach described in the work of Pernici et al. [46], in which features are learned to follow a set of special fixed classifier prototypes. Pernici et al. [46] impose global stationarity using a classifier in which prototypes cannot be trained (i.e., fixed) and are set before training. Under this condition, only the direction of the features aligns toward the fixed directions of the classifier prototypes and not the opposite. This constraint imposes learned features to follow their corresponding fixed prototypes, therefore encouraging representation stationarity. The lack of trainable classifier functionality is basically replaced by previous layers. Fixed prototypes are set according to the coordinate vertices of a d-Simplex regular polytope that, in addition to stationarity, allows maximally separated features to be learned [44, 45].

We take advantage of this result and perform CiL as a surrogate task to learn stationary features’ representation to achieve compatibility. More formally, let \(\mathbf {W} \; \forall t \in \lbrace 1, 2, \ldots , T\rbrace\) be the d-Simplex fixed classifier, and we instantiate the CiL problem as \(\sigma (\phi _t \circ \mathbf {W})\) , where \(\sigma\) indicates the softmax function, and perform learning according to incremental fine-tuning. The evolving training-set \(\mathcal {T}_t \leftarrow \mathcal {M}_{t} \cup \mathcal {D}_t\) is computed according to a rehearsal base strategy using the episodic memory, \(\mathcal {M}_{t}\) , which contains an updating set of samples from \(\lbrace \mathcal {D}_1, \ldots , \mathcal {D}_{t-1} \rbrace\) . The memory is updated as \(\mathcal {M}_{t+1} \leftarrow \mathcal {M}_{t} \cup {\rm S}{\rm\small{AMPLING}}({\rm }D_{t})\) . The loss optimized in the work of Pernici et al. [46] is adapted to CL²R training as follows:where \(K_s\) is the set of classes learned up to time t, \(|\mathcal {T}_{t}|\) is the number of elements in the training-set, \(K_u\) is the set of the outputs of the classifier that have not yet been assigned to classes at time t (i.e., future unseen classes [47]), \(\mathbf {w}^{\top }_{(\cdot)}\) is a class prototype of the fixed classifier \(\mathbf {W}\) , and \(y_i\) is the supervising label. In particular, \(\mathbf {W}\) is the weight matrix of the fixed classifier, which does not undergo learning during model training. In the work of Pernici et al. [46], the d-Simplex prototypes are defined as \(\mathbf {W} = \lbrace e_1,e_2,\dots ,e_{d-1}, \alpha \sum _{i=1}^{d-1} e_i \rbrace ,\) where d is the feature dimensionality of the d-Simplex, \(\alpha =\frac{1-\sqrt {d+1}}{d}\) , and \(e_i\) denotes the standard basis in \(\mathbb {R}^{d-1}\) , with \(i \in \lbrace 1,2, \dots , d-1\rbrace\) .

The loss of Equation (7) imposes global stationarity and does not require any knowledge to be extracted from the previously learned models. However, catastrophic forgetting causes misalignment between features and fixed classifier prototypes. Therefore, we further impose additional stationarity constraints in a local neighborhood of a feature by encouraging the current model to mimic the feature representation of the model previously learned. This allows the overall stationarity to also be determined by a local learning mechanism interacting with the global one provided by the d-Simplex classifier of Equation (7). The global-to-local interaction is achieved through the FD loss [56]. Differently from the more common practice of FD in CiL [16, 22, 26] in which each mini-batch is sampled from both the episodic memory and the current task (i.e., \(\mathcal {T}_t \leftarrow \mathcal {M}_{t} \cup \mathcal {D}_t\) ), we evaluate the FD loss, at each task t, only on the samples stored in episodic memory \(\mathcal {M}_t\) observed from previous tasks:where \(\phi _{t-1}\) is the model learned from the previous task. This encourages local stationarity and stability from only the previous classes in the episodic memory and the assimilation of new knowledge (plasticity) from only the classes of the current task. As confirmed by ablation in Section 8, this learning strategy leads to a significant performance improvement. The final optimized loss function is the sum of Equations (8) and (7):where \(\lambda\) balances the contribution of global and local alignment provided by the two losses. The pseudo-code in Algorithm 1 and in Algorithm 2 detail our training procedure and its application in visual search, respectively.

We analyze by ablation the main components of our training procedure. The ablation is performed on the CIFAR100 dataset as described in Section 7.4 and considers the 10-task CL²R setting, which can be regarded as a worst-case scenario for this dataset. We analyze the impact of (i) the specific classifier: Trainable vs. fixed d-Simplex with or without the FD component, (ii) how the FD loss is evaluated, and (iii) the sensitivity of the number of samples reserved per class in the episodic memory.

Impact of the d-Simplex fixed classifier and FD. As can be noticed from Table 9, the Trainable classifier is not able to learn compatible representations. When combined with FD, the performance improves only marginally and not sufficiently to be compared with the CiL approaches shown in Table 3. FD evaluated on the only samples stored in episodic memory as defined in \(\mathcal {L}_{\scriptscriptstyle \textrm {FD}}^{\scriptscriptstyle \mathcal {M}}\) (Equation (8)) improves the values of the reported metrics showing a better supervision signal for the updated model. The d-Simplex alone improves on the previous components obtaining values of AC = 0.27 and FC = 0.003, which are higher than the Trainable classifier with \(\mathcal {L}_{\scriptscriptstyle \textrm {FD}}^{\scriptscriptstyle \mathcal {M}}\) . This remarks on the importance of preserving the global geometry of the learned features according to the d-Simplex fixed classifier.

Impact of memory samples on FD. Table 9 shows that when the distillation loss is evaluated on the only samples stored in episodic memory \(\mathcal {L}_{\scriptscriptstyle \textrm {FD}}^{\scriptscriptstyle \mathcal {M}}\) (Equation (8)), our approach achieves better overall results. We argue that this positive effect is mostly due to the interaction between the global feature stationarity learned using the fixed classifier and the local one promoted through FD from the only observed samples in the episodic memory. The interaction is most likely related to the fact that the fixed d-Simplex classifier in general does not allow novel classes to interfere in the feature space of the already learned one. This in turn provides favorable working conditions (i.e., a kind of coarse pre-alignment) for achieving feature alignment with respect to the previous model by the distillation loss. As expected, the impact intensifies when evaluated only on already known classes, as alignment is less prone to unexpected noisy features which may reduce the degree of the alignment. This confirms the effectiveness of restricting FD only on memory samples in contrast to the traditional FD commonly used in CiL.

Impact of the episodic memory size. Figure 3 shows the effect of different numbers of reserved samples per class for both our learning procedure and other baselines. As expected, the more samples per class are reserved in the episodic memory, the better the performance. Our approach, with 20 samples per class, achieves results similar to those obtained by the other methods with more examples per class. Although ER, LUCIR, and FAN have a better relative improvement with 50 samples per class, overall our approach results in the highest performance in learning compatible features.

We also evaluated the methods in the challenging memory-free training setting (i.e., without the episodic memory). Our training procedure achieves the highest results also in this condition, remarking on the fact that CiL methods typically do not have an inherent mechanism to learn compatible features.

In this article, we have introduced the problem of CL²R, which considers the compatibility learning problem within the lifelong learning paradigm. We introduced a novel set of metrics to properly evaluate this problem and proposed a novel CL²R training procedure that imposes global and local stationarity on the learned features to achieve compatibility between representations under catastrophic forgetting. Global and local stationarity is imposed according to the d-Simplex fixed classifier and the FD loss, respectively. Empirical evaluation of the learned lifelong-compatible representation shows the effectiveness of our method with respect to baselines and state-of-the-art methods.