Modular Dimensionality Reduction

Abstract

We introduce an approach to modular dimensionality reduction, allowing efficient learning of multiple complementary representations of the same object. Modules are trained by optimising an unsupervised cost function which balances two competing goals: Maintaining the inner product structure within the original space, and encouraging structural diversity between complementary representations. We derive an efficient learning algorithm which outperforms gradient based approaches without the need to choose a learning rate. We also demonstrate an intriguing connection with Dropout. Empirical results demonstrate the efficacy of the method for image retrieval and classification.

1 Introduction

High dimensional data is a widespread challenge in machine learning applications, from computer vision through to bioinformatics and natural language processing. A natural solution is to find a structure-preserving mapping to a low dimensional space, many techniques for which can be found in the literature, such as kernel PCA, Isomap, LLE and Laplacian Eigenmaps [6, 23]. This paper provides a meta-level tool for modular dimensionality reduction, applicable to each of the aforementioned approaches.

We start from the observation that multiple abstractions of the same concept can be taken, and may provide complementary views on a task of interest. We therefore propose a modular approach to unsupervised dimensionality reduction, in which we learn a diverse collection of low-dimensional representations of the data. Once a modular representation is learned, each module may be used independently – with their respective predictions combined at test time. This procedure is naturally parallelisable in a distributed computing architecture; and, since each representation is low-dimensional, processing for each module is fast and efficient.

In the context of supervised learning, successful ensemble performance emanates from a fruitful trade-off between the accuracy of the individual members of the ensemble and the degree of diversity [4, 15]. We carry this insight across to the domain of unsupervised dimensionality reduction, by demonstrating the importance of diversity for a set of representation modules. We introduce an unsupervised loss function for training a set of dimensionality reduction modules, which balances two competing objectives. The first objective is for each module to preserve relational structure within the original feature space; the second is for modules to exhibit a diversity of relational structures.

The contributions of this paper are as follows:

• 1. An unsupervised loss function for modular dimensionality reduction.

• 2. A bespoke optimisation procedure which outperforms gradient based methods such as stochastic gradient descent in our setting.

• 3. A detailed empirical comparison with competitors.

• 4. An intriguing connection to the dropout algorithm from deep learning [13].

2 Background

We first review work on dimensionality reduction and ensemble learning.

2.1 Unsupervised Dimensionality Reduction

The canonical approach to unsupervised dimensionality reduction is PCA, and its kernelised generalisation, KPCA [21]. KPCA is a general approach which may be applied to a wide variety of application domains through an appropriate choice of kernel [19, 25]. Several manifold learning techniques have also been shown to be special cases of KPCA, with a data-dependent kernel [12].

Classically, KPCA has been viewed as the orthogonal projection which maximises the preserved variance [21]. We shall adopt an alternative perspective in which we view KPCA as a form of unsupervised similarity learning, whereby a mapping is chosen so that inner-products in the low dimensional space approximate the kernel. To make this precise we require some notation. We let $\mathcal {X}$ denote our original feature space and let $k:\mathcal {X}\times \mathcal {X}\rightarrow \mathbb {R}$ denote a symmetric positive semi-definite kernel function. Take an unlabelled dataset $\mathcal {D}= \left\{ \varvec{x}_1,...,\varvec{x}_N\right\} \subset \mathcal {X}$. For simplicity, we assume throughout that k is centred with respect to $\mathcal {D}$ [21]. Let $\mathbb {H}_k$ denote the associated reproducing kernel Hilbert space (RKHS) of real-valued functions. For each $H\in \mathbb {N}$, we let $\mathbb {H}_{k}^H$ denote the class of H-dimensional mappings $\varvec{\varphi }:\mathcal {X}\rightarrow \mathbb {R}^H$ with coordinate functions taken from the RKHS $\mathbb {H}_{k}$. That is, for each $\varphi \in \mathbb {H}_k^H$ there exists $\varphi ^1,\cdots ,\varphi ^H\in \mathbb {H}_{k}$ such that for all $x\in \mathcal {X}$, $\varphi (x)=\left( \varphi ^h(x)\right) _{h=1}^H$.

Definition 1

(Inner product loss function). Given an unsupervised data set $\mathcal {D}$ and a mapping $\varvec{\varphi }\in \mathbb {H}_k^H$, the inner product loss is given by

$$\begin{aligned} L_k\left( \varvec{\varphi },\mathcal {D}\right) = \frac{1}{N^2}\sum _{ i,j\le N} \left( \langle \varvec{\varphi }\left( \varvec{x}_i \right) , \varvec{\varphi }\left( \varvec{x}_j \right) \rangle -k\left( \varvec{x}_i,\varvec{x}_j\right) \right) ^2. \end{aligned}$$

We can interpret KPCA as a form of unsupervised similarity learning which minimises the inner product loss. Let $\xi :\mathcal {X}\rightarrow \mathbb {H}_k$ denote the canonical embedding given by $\xi (x)(y) = k(x,y)$.

Proposition 1

The inner product loss $L_k\left( \varvec{\varphi },\mathcal {D}\right)$ is minimised by taking $\varvec{\varphi }$ to be the member of $\mathbb {H}_{k}^H$ obtained by embedding $\mathcal {D}$ into $\mathbb {H}_k$ via $\xi$ and projecting onto the top H kernel principal components.

The proofs of all results within the main text are given in the appendices (see supplementary material).

2.2 Ensembles and Diversity

Combining the outputs of multiple predictors often brings both statistical advantages, such as bias or variance reduction, and computational advantages, through parallelism. In order outperform an individual model, ensembles promote a level of diversity or disagreement between the predictions the constituent models [10, 15]. Whilst methods such as bagging and boosting encourage diversity through a manipulation of the training data, a more direct approach is the Negative Correlation Learning (NCL) algorithm of Liu and Yao [18] in which diversity is targeted explicitly.

Suppose we have a supervised regression ensemble $\mathcal {H}=\left\{ h_m\right\} _{m=1}^M$ consisting of predictors $h_m$. In the previous section we used an unsupervised dataset $\mathcal {D}=\{\mathbf{x}_1,...,\mathbf{x}_N\}$. To distinguish this we use notation $\mathcal {T}=\{(\mathbf{x}_1,y_1),...,(\mathbf{x}_N,y_N)\}$ for a supervised dataset. We let $\mathbb {V}(\cdot )$ denote the empirical variance of a finite sequence. The NCL algorithm can be understood in terms of the following modular loss function.

Definition 2

(Modular loss function). The modular loss $E_{\lambda }$ is defined by

$$\begin{aligned} E_{\lambda }(\mathcal {H},\mathcal {T})= \frac{1}{NM}\sum _{n=1}^N\sum _{m=1}^M {{\left( h_m(\varvec{x}_n)-y_n\right) ^2}}-\lambda \cdot \frac{1}{N}\sum _{n=1}^N \mathbb {V}\left( \left\{ h_m(\varvec{x}_n)\right\} _{m=1}^M\right) . \end{aligned}$$

The modular loss function consists of two terms: A squared loss term which targets the average individual accuracy of the predictors $h_m$, combined with a diversity term which encourages disagreement between the predictors. The hyper-parameter $\lambda$ controls the degree of emphasis placed on the diversity. This has the special property that when $\lambda =1$, $E_{\lambda }(\mathcal {H},\mathcal {T})$ is exactly the squared loss for the ensemble predictor $\frac{1}{M}\sum _m h_m(\mathbf{x})$ from the target y.

The NCL algorithm is equivalent to stochastic gradient descent applied to the modular loss. This perspective differs from original formulation of the NCL algorithm first introduced by Liu and Yao which utilises a multiplicity of interacting cost functions [18]. However, the updates of the two formulations are equal up to a factor of 1/M applied to the learning rate.

3 The Modular Inner Product Loss

Our goal is to train a collection of M distinct but complementary representations of the data. With this goal in mind, we introduce the modular inner product loss which combines two contrasting objectives. On the one hand, we seek high quality representations which faithfully preserve the relational structure encoded by the kernel. On the other hand, we would like the relational structure encoded in our different representations to be diverse. Let $\mathcal {F}(H,M)$ denote the class of all M-tuples $\varvec{\varPhi }=\left\{ \varvec{\varphi }_m\right\} _{m=1}^M$ with each $\varvec{\varphi }_m\in \mathbb {H}_{k}^H$. Recall that $\mathbb {V}(\cdot )$ denotes the empirical variance.

Definition 3

(The modular inner product loss). Suppose we have an unlabelled data set $\mathcal {D}\subset \mathcal {X}$ and a kernel k. Given $\varvec{\varPhi } \in \mathcal {F}(H,M)$, the modular inner product loss is given by

$$\begin{aligned} L_k^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right) = \frac{1}{M}\sum _{m=1}^M L_k\left( \varvec{\varphi }_m,\mathcal {D}\right) -\lambda \cdot \frac{1}{N^2}\sum _{1\le i.j\le N}\mathbb {V}\left( \left\{ \varphi _m(\varvec{x}_i)\cdot \varphi _m(\varvec{x}_j)\right\} _{m=1}^M\right) . \end{aligned}$$

(1)

The modular inner product loss is an analogue of the supervised modular loss function (Definition 2), with inner products between a pair of examples in a representation module replacing predictions for a single example, and the target replaced by an unsupervised inner product.

An equivalent reformulation of the modular inner product loss is as a convex combination between the average inner product loss of the individual modules and the inner product loss of a composite representation. Given $\varvec{\varPhi }\in \mathcal {F}\left( H,M\right)$ we define $\overline{\varvec{\varPhi }}\in \left( \mathbb {H}_{k}\right) ^{H\cdot M}$ by $\overline{\varvec{\varPhi }}(\varvec{x}) =\left( 1/\sqrt{M}\right) \cdot \left[ \varvec{\varphi }_1(\varvec{x})^T,\cdots ,\varvec{\varphi }_M(\varvec{x})^T\right] ^T$. Proposition 2 is proved in Appendix 9 (see supplementary material).

Proposition 2

$L_k^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right) =(1-\lambda )\cdot \frac{1}{M}\sum _{m=1}^M L_k\left( \varvec{\varphi }_m,\mathcal {D}\right) +\lambda \cdot {L}_{k}\left( \overline{\varvec{\varPhi }},\mathcal {D}\right)$.

When $\lambda = 0$ the loss $L_k^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right)$ is minimised by taking each $\varvec{\varphi }_m$ to be a projection onto the top H kernel principal components, whilst for $\lambda = 1$, $L_k^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right)$ is minimised by taking $\overline{\varvec{\varPhi }}$ to be the projection onto the top $M\cdot H$ kernel principal components. Hence, $L_k^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right)$ blends smoothly between training representation modules as individuals and targeting the composite representation.

4 Efficient Optimization

We now introduce the module-by-module (MBM) algorithm, which is a form of alternating optimisation designed to minimise the modular inner product loss without the need to choose a learning rate. Our objective is to minimise $L_{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right)$ over $\varvec{\varPhi }\in \mathcal {F}\left( H,M\right)$. We require an empirical kernel map.

Definition 4

A rank R empirical kernel map is a function $\varvec{\psi }\in \mathbb {H}_k^R$ such that $\varvec{\psi }(\varvec{x}_i)^T\psi (\varvec{x}_j) = k(\varvec{x}_i,\varvec{x}_j)$ for all pairs $(\varvec{x}_i,\varvec{x}_j)\in \mathcal {D}^2$.

One can always construct an empirical kernel map of rank N by taking $\varvec{\psi }(\varvec{x}) = \varvec{K}(\mathcal {D})^{-\frac{1}{2}}\left[ k(\varvec{x},\varvec{x}_1),\cdots ,k(\varvec{x},\varvec{x}_N)\right] ^T$, where $\varvec{K}(\mathcal {D})=\left( k(\varvec{x}_i,\varvec{x}_j)\right) _{ij}$ denotes the kernel gram matrix. Moreover, given a kernel k we can often obtain a low rank empirical kernel map $\psi$ for a kernel $\tilde{k}$ which closely approximates k by employing a method such as random Fourier features [11] or the Nyström method [27]. By reasoning analogous to [20] we have the following useful proposition.

Proposition 3

Given a rank R empirical kernel map $\psi$, the minimum for $L_k^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right)$ is attained by $\varvec{\varPhi } = \left\{ \varvec{\varphi }_m\right\} _{m=1}^M$ with each $\varvec{\varphi }_m$ of the form $\varvec{\varphi }_m( \varvec{x}) = \varvec{W}_m\cdot \varvec{\psi }(\varvec{x})$ for some matrix $\varvec{W}_m \in \mathbb {R}^{H\times R}$.

Hence, our objective reduces to the following matrix optimisation problem: Minimise

$$\begin{aligned} C^{\lambda }\left( \mathcal {W},\varvec{\varPsi }\right) :&=\sum _{m=1}^M\left| \left| \varvec{F}_m^T\varvec{F}_m - \varvec{\varPsi }^T\varvec{\varPsi }\right| \right| ^2-\lambda \cdot \sum _{m=1}^M\left| \left| \varvec{F}_m^T\varvec{F}_m-\frac{1}{M} \sum _{q=1}^M\varvec{F}_q^T\varvec{F}_q \right| \right| ^2\\ {}&\propto L^{\lambda }_k\left( \varvec{\varPhi _{\mathcal {W}}},\mathcal {D}\right) , \end{aligned}$$

where $\varvec{\varPsi }=\left[ \varvec{\psi }(\varvec{x}_1),\cdots ,\varvec{\psi }(\varvec{x}_n)\right] \in \mathbb {R}^{R\times N}$, $\varvec{F}_m = \varvec{W}_m\cdot \varvec{\varPsi }$ and $\varvec{\varPhi _{\mathcal {W}}}=\left\{ \varvec{W}_m\cdot \varvec{\psi }\right\} _{m=1}^M$. We make use the concept of the rank-constrained approximate square root of a symmetric matrix.

Definition 5

Define $RT_r:\mathbb {R}^{d\times d}\rightarrow \mathbb {R}^{r\times d}$ by

$$RT_r(\varvec{M}) = {argmin}_{\varvec{F} \in \mathbb {R}^{r\times d}}\left\{ \left| \left| \varvec{F}^T\varvec{F}-\varvec{M}\right| \right| ^2\right\} .$$

Dax has shown that the rank-constrained approximate square root $RT_r(\varvec{M})$ of any $d\times d$ symmetric matrix $\varvec{M}$ (not necessarily positive semi-definite) may be computed via the singular value decomposition in $O(d^2\cdot r)$ time and $O(d^2)$ space complexity [7]. The following proposition allows us to optimise the weights of a single module $\varvec{\varphi }_m$, whilst leaving the remaining modules fixed.

Proposition 4

Suppose we take $m \in \left\{ 1,\cdots ,M\right\}$, fix $\varvec{W}_q$ for $q \ne m$, and let

$$\begin{aligned} \varvec{T}_m = \frac{M}{M-\lambda \cdot (M-1)} \cdot \varvec{\varPsi }^T\left( \varvec{I}_D-\frac{\lambda }{M}\cdot \sum _{q \ne m}\varvec{W}_q^T\varvec{W}_q\right) \varvec{\varPsi }. \end{aligned}$$

Take $\varvec{F}_m = RT_H\left( \varvec{T}_m\right)$. Setting $\varvec{W}_m = \varvec{F}_m\varvec{\varPsi }^{\dagger }$ minimises $C^{\lambda }\left( \mathcal {W},\varvec{\varPsi }\right)$ with respect to $\varvec{W}_m$, under the constraint that $\varvec{W}_q$ remains fixed for $q \ne m$, where $\varvec{\varPsi }^{\dagger }$ denotes the psuedo-inverse of $\varvec{\varPsi }$.

Unfortunately, computing $\varvec{F}_m$ via Proposition 4 is $O(N^2\cdot H)$ which is intractable for large N. The following proposition enables us to reduce the complexity of this optimisation whenever we have access to an empirical kernel map of rank $R \ll N$.

Proposition 5

Suppose that $\varvec{\psi }$ is an empirical kernel map of rank R. Take $\tilde{\varvec{\varPsi }}=(RT_{R}(\varvec{\varPsi }\varvec{\varPsi }^T))^T \in \mathbb {R}^{R \times R}$. For all $\mathcal {W} =\left\{ \varvec{W}_m\right\} _{m=1}^M$ with $\varvec{W}_{m}\in \mathbb {R}^{H\times R}$ we have $C_{\lambda }\left( \mathcal {W},\tilde{\varvec{\varPsi }}\right) = C_{\lambda }\left( \mathcal {W},{\varvec{\varPsi }}\right)$. Moreover, computing $\tilde{\varvec{\varPsi }}$ is $O\left( R^2\cdot N\right)$ in time complexity and $O(R^2)$ in space complexity.

Combining Propositions 3, 4 and 5 gives rise to the module-by-module algorithm (MBM, Algorithm 1), which is $O(N R^2+EH R^2)$ in time and O(NR) in space complexity, and has the advantage of reducing the modular inner product loss at every iteration until a critical point is reached.

The following theorem justifies the use of the MBM algorithm - it is guaranteed to reduce the modular inner product loss at every epoch until a critical point is reached.

Theorem 1

Given $E \in \mathbb {N}$, let $\varvec{\varPhi }^E \in \mathcal {F}(H,M)$ denote the set obtained by training with Algorithm 1, for E epochs. Then for all $E \in \mathbb {N}$, $L_k^{\lambda }\left( \varvec{\varPhi }^{E+1},\mathcal {D}\right) < L_k^{\lambda }\left( \varvec{\varPhi }^E,\mathcal {D}\right)$, unless $\varvec{\varPhi }^E$ is a critical point of $L_k^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right)$, in which case $L_k^{\lambda }\left( \varvec{\varPhi }^{E+1},\mathcal {D}\right) \le L_k^{\lambda }\left( \varvec{\varPhi }^E,\mathcal {D}\right)$.

5 The Dropout Connection

In this section we introduce a surprising connection between the modular inner product loss and the dropout algorithm [13, 22]. Dropout is a state of the art approach to regularising deep neural networks in which a random collection of hidden neurons is “dropped out” at each stochastic gradient update. The dropout algorithm can be understood as implicitly minimising the expectation of a stochastic loss function based on predictions from a random sub-network [22, 26]. There is a natural analogue of this, in our setting: to minimise the expectation of a stochastic variant of the inner product loss, based on inner products computed from a random subset of modules. We refer to this analogue as the drop-module (DM) algorithm. To be precise, given an ensemble of feature mappings $\varvec{\varPhi }\in \mathcal {F}(H,M)$ each binary vector $\varvec{\eta } = \left\{ \eta _m \right\} _{m=1}^M \in \left\{ 0,1\right\} ^M$ corresponds to a ‘noisy’ representation $\varvec{\varPhi }_{\varvec{\eta }}$ given by $\varvec{\varPhi }_{\varvec{\eta }}(\varvec{x}) =({1}/{\sqrt{M}})\cdot \left( \eta _1\cdot \varvec{\varphi }_1(\varvec{x}),\cdots ,\eta _M\cdot \varvec{\varphi }_M(\varvec{x})\right)$.

Fix a probability $p \in [0,1]$ and let B(p) denote the probability measure on $\{0,1\}$ with $\mathbb {E}_{B(p)}(\eta )=p$. Let $\varvec{\varvec{\varTheta }}$ denote the parameters of $\varvec{\varPhi }$. The DM algorithm proceeds by randomly sampling $\varvec{x}_i,\varvec{x}_j \in \mathcal {D}$ and $\eta _m \sim B(p)$ and updating

$$ \varvec{\varvec{\varTheta }} \leftarrow \varvec{\varvec{\varTheta }}-\alpha \cdot \frac{\partial }{\partial \varvec{\varvec{\varTheta }}}\left( \langle \varvec{\varPhi }_{\varvec{\eta }}\left( \varvec{x}_i \right) , \varvec{\varPhi }_{\varvec{\eta }}\left( \varvec{x}_j \right) \rangle -k\left( \varvec{x}_i,\varvec{x}_j\right) \right) ^2. $$

The DM algorithm implicitly minimises the following stochastic loss function

$$ L_{k,p}^{\text {drop}}\left( \varvec{\varPhi },\mathcal {D}\right) =\mathbb {E}_{\varvec{\eta }\sim B(p)^M}\left[ L_k\left( \varvec{\varPhi }_{\varvec{\eta }},\mathcal {D}\right) \right] . $$

Previously, Baldi et al. demonstrated that dropout may be understood as training an exponentially large ensemble with shared weights [1]. In our setting this corresponds to shared weight a ensemble of size $2^M$ with an ensemble member for each $\varvec{\eta }\in \{0,1\}^M$. We demonstrate that the DM algorithm can be related to an ensemble of size M, trained via the modular inner product loss (see Definition 3). We emphasise that unlike the shared weight ensembles considered by Baldi et al. [1], here we consider ensembles with separate weights in which the interaction takes place purely via the diversity term in the modular inner product loss.

Theorem 2

The drop-module inner product loss at p is equivalent to the modular inner product loss at $\lambda = Mp/(1+p(M-1))$. To be precise, for $\varvec{\varPhi }_0\in \mathcal {F}(H,M)$ we have

$$\begin{aligned} \frac{\partial L_{k,p}^{\text {drop}}\left( \varvec{\varPhi },\mathcal {D}\right) }{\partial \varvec{\varPhi }}\bigg |_{\varvec{\varPhi }=\varvec{\varPhi }_0} = p\cdot \frac{\partial L_{k}^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right) }{\partial \varvec{\varPhi }}\bigg |_{\varvec{\varPhi }=(\sqrt{p/\lambda })\cdot \varvec{\varPhi }_0}. \end{aligned}$$

Theorem 2 implies that if we take $\lambda = Mp/(1+p(M-1))$ then the minima of $L_{k}^{\lambda }\left( \varvec{\varPhi },\mathcal {D}\right)$ are equal to the minima of $L_{k,p}^{\text {drop}}\left( \varvec{\varPhi },\mathcal {D}\right)$, up to a constant scaling factor of $\sqrt{p/\lambda }$. In this sense, the minima for the two loss functions are representationally equivalent. The relationship between the diversity parameter $\lambda$ in the MBM algorithm and the probability of keeping module p in the DM algorithm is illustrated in Fig. 1.

Fig. 1.

The diversity parameter $\lambda$ in MBM vs. the corresponding p in DM.

6 Experimental Results

In this section we first demonstrate the optimisation performance of the MBM algorithm before comparing our method for other natural approaches for training multiple kernelised representations. The data sets used in all experiments are described in Sect. 12.1 (see supplementary material).

Table 1. A comparison of the MBM algorithm with gradient based methods.

6.1 Optimisation Performance of the MBM Algorithm

In this section we assess the MBM algorithm (Algorithm 1) in terms of its efficiency at optimising the modular inner product loss. We compare with two gradient based approaches. As a baseline we consider stochastic gradient descent (SGD) applied directly to the modular inner product loss. This is expected to perform poorly in our setting since the modular inner product loss sums over all pairs of examples. We also consider the state of the art Adam optimiser of Kingma et al. [14] (Adam). The Adam optimiser is applied in batch mode, first compressing the data by applying Proposition 5. We set $M=10$, $H=10$, $\lambda =0.9$ and let k to be the Gaussian kernel with $\gamma$ set using the heuristic of Kwok and Tsang [16, Sect. 4]. In addition, we employ a rank $R=1000$ Nyström approximation [27]. For SGD and Adam we consider learning rates in the range $\{10^{-6},\cdots ,10^2\}$. We evaluated the algorithms by training for one hour and evaluating the minimum value of the loss function attained during training and the convergence time - the time taken for the loss function to fall within $1\%$ of its minimum. For SGD and Adam we report results corresponding to the learning rate which achieves the lowest minimum loss. The results are shown in Table 1. The SGD method was extremely slow and typically failed to converge within one hour. The compressed Adam method performed much better and typically converged within 30 min. However, the bespoke MBM algorithm achieved the same minimum loss in at most twice the speed on each of the data sets. The MBM algorithm also has the advantage of not requiring the user to set a learning rate.

6.2 Image Retrieval and Classification Performance of MBM Modules

We compare four unsupervised approaches to training multiple kernelised feature mappings:

Fig. 2.

Test time as vs. H. See Appendix 12.2 for discussion & other data sets (see supplementary material).

Partition. We compute the top HM KPCAs and randomly partition these into M sets of H, so that each mapping $\varvec{\varphi }_m\in \mathbb {H}_k^H$ is a projection onto a disjoint subset of the top $H\cdot M$ KPCAs.

Bootstrap. Bagging [3] applied to KPCA. For each $m \in \left\{ 1,\cdots ,M\right\}$ take a bootstrap sample $\tilde{\mathcal {D}}_m$, of size N, and let $\varvec{\varphi }_m\in \mathbb {H}_k^H$ be the KPCA projection mapping onto H dimensions for $\tilde{\mathcal {D}}_m$.

Random. A kernelised variant of the widely used technique of random projections [2, 8]. For each $m=1,\cdots ,M$ we sample a random matrix $\varvec{R}_m \in \mathbb {R}^{H\times D}$ from an $H\times D$ standard normal distribution, and normalise each row so that it has unit norm. In order to kernelise this technique the feature space for the random matrices is the output of the empirical kernel map $\psi$ (see Sect. 4).

MBM. Our proposed approach in which $\varvec{\varPhi }$ is trained to minimise the modular inner product loss via MBM algorithm. The diversity parameter $\lambda$ is set based upon performance on a validation set. We shall consider $H\in \left\{ 5, 10, 15, 20, 30, 50, 100\right\}$ and take M so that $H\cdot M=300$. We also compare with the following non-modular base line.

Monolithic. A single mapping $\varvec{\varphi }\in \mathbb {H}_k^H$ - the projection onto the top 300 KPCAs.

In each case we take k to be the Gaussian kernel with the $\gamma$ parameter set via the heuristic of Kwok and Tsang [16, Sect. 4]. For computational efficiency we employ a rank 1000 Nyström approximation [27] in each case. We shall consider two distinct tasks:

Image Retrieval: We shall consider the modular low dimensional representation’s capability for efficiently retrieving a set of $\kappa$ close-by images. Let $\varvec{\varPhi } = \left\{ \varvec{\varphi }_m\right\} _{m=1}^M \in \mathcal {F}(H,M)$ be a modular representation and $\mathcal {D}$ an unlabelled training set. Given a test point $\varvec{x}\in \mathcal {X}$, for each module, we compute the set $\mathcal {I}^{\varphi _m}_{\kappa ,n}(\varvec{x})\subset \mathcal {D}$ of $\kappa$-nearest neighbours of $\varvec{x}$ based the distance $\Vert \varphi _m(\varvec{x}_q)-\varphi _m(\varvec{x})\Vert _2$. We then extract subset of size $\kappa$, $\mathcal {I}^{\varvec{\varPhi }}_{\kappa ,n}(\varvec{x})\subset \bigcup _{m=1}^M \mathcal {I}^{\varphi _m}_{\kappa ,n}(\varvec{x})$ so that the elements $\varvec{x}_q\in \mathcal {I}_{\kappa ,n}^{\varvec{\varPhi }}(\varvec{x})$ minimise the average squared distance from the test point $\varvec{x}$ over the low dimensional spaces ie. $(1/M)\cdot \sum _{m=1}^M \Vert \varphi _m(\varvec{x}_q)-\varphi _m(\varvec{x})\Vert _2^2$ is minimised. Let $\mathcal {I}_{{\kappa },n}(\varvec{x})$ denote the set of ${\kappa }$ nearest neighbours as computed in the original space $\mathcal {X}$. To assess performance we compute the precision: the average value of $(1/\kappa )\cdot \#\left( \mathcal {I}^{\varvec{\varPhi }}_{\kappa ,n}(\varvec{x})\cap \mathcal {I}_{{\kappa },n}(\varvec{x})\right)$. This procedure is based upon the method of [24] and gives a quantitative assessment of the representation’s ability to preserve structural information. The results of the image retrieval task for $\kappa =10$, $H=20$ and $M=15$ are shown in Table 2. On each of the eight data sets the precision attained by the MBM method significantly exceeds the precision attained by the other modular methods: partition, bootstrap and random. Table 3 compares MBM with the monolithic method in which we simply compute the 10 nearest neighbours in the $\varphi$-projected space, where $\varphi$ is the projection onto the top 300 KPCAs. For a relatively modest reduction in performance the MBM method obtains a significant speed up at test time. The speed up is due to the fact that each set of nearest neighbours $\mathcal {I}^{\varphi _m}_{\kappa ,n}(\varvec{x})$ may be computed in parallel on a low-dimensional space (see Fig. 2 and Appendix 12.2 (see supplementary material). Figure 3 demonstrates the precision as a function of the number of dimensions per module (H) with $\kappa =10$ and $H\cdot M=300$. As H increases, the precision approaches the precision attained by the 300-dimensional Monolithic approach. The precision attained by the MBM approach typically exceeds that attained by the other modular approaches (Bootstrap, Partition, Random) across a range of values of H. Corresponding figures for other data sets are given in Appendix 12.3 (see supplementary material).

Fig. 3.

Precision as a function of H (See Sect. 6.2).

Table 2. A comparison of methods for modular dimensionality reduction. See Sect. 6.2 for details.

Table 3. Comparing the MBM & the Monolithic approach (See Sect. 6.2).

Fig. 4.

Classification accuracy as a function of H (See Sect. 6.2).

Fig. 5.

Performance as a function of the diversity parameter ($\lambda$) (see Sect. 6.2).

Classification. We compare the methods in terms of their capacity for extracting multiple sets of features for use in a classification ensemble. Given a modular representation $\varvec{\varPhi } = \left\{ \varvec{\varphi }_m\right\} _{m=1}^M \in \mathcal {F}(H,M)$, for each m we train a classifier $f_m$ based on the features extracted by $\varphi _m$. Given a test point $\varvec{x}$ we combine the outputs of $\left\{ f_m\left( \varphi _m(\varvec{x})\right) \right\} _{m=1}^M$ by taking a modal average. Table 2 shows the classification accuracy for ensembles consisting of 15 5-nearest neighbour classifiers trained on 20-dimensional spaces. The MBM approach significantly outperforms the other approaches on five out of eight data sets, and performs comparably or better than the alternatives on every data set. Table 3 compares with the monolithic approach - a single 5-nearest neighbour classifier on 300 KPCAs. The MBM approach is both faster and more accurate than the monolithic method on all but one data set (Fig. 4).

The Diversity Parameter. The diversity parameter $\lambda$ in the MBM algorithm controls the level of emphasis placed upon encouraging a diversity of representations. We found that the optimal performance (both in terms of information retrieval and classification) was typically attained with $\lambda$ just below 1, with performance declining sharply by taking $\lambda =1$ (see Fig. 5, cols 1& 2). It is interesting to observe that the dropout algorithm often performs well with $p\approx 0.5$ and this corresponds a value of $\lambda$ just below 1, when M is large (see Fig. 1). However, whilst this pattern was observed on all data sets for image retrieval (see Appendix 12.6) (see supplementary material), for some data sets the best classification performance was attained by taking much lower values of $\lambda$ (see Fig. 5, col 3 and Appendix 12.6 (see supplementary material). Ultimately, the optimal value of $\lambda$ is data dependent and must be set based on validation performance.

7 Discussion

We have investigated a method for modular unsupervised dimensionality reduction. Our method is based upon the modular inner product loss (Definition 3), an adaptation of concepts from both negative correlation learning [4, 18] and kernel principal components analysis [21]. Whilst the modular loss could be optimised by gradient based methods we introduced a novel module-by-module algorithm, which converges at least twice as fast as a state of the art gradient based optimiser [14] without the need to tune the learning rate.

Modular representations have the potential to be applied on range of tasks. Empirical results on both image retrieval and classification tasks confirm that the MBM algorithm is superior to a range of competitors including random projections and bootstrapping, whilst providing a parallelisation advantage over “monolithic” dimensionality reduction. We also demonstrated an intriguing equivalency between our proposal and an analogue of the dropout algorithm - drop module, which deserves further attention.

In summary, this work has shown the potential of explicitly managing diversity in unsupervised representation learning.