Robust Multi-view Common Component Learning

Abstract

In many computer vision applications, one object usually exists more than one data representation. This paper focuses on the specific problem of cross-view recognition, which aims to recognize objects from different views. A majority of representative works mainly attempt to seek a common subspace, in which the Euclidean distance of within-class data is short. Intuitively, the recognition performance will be better if the various data from the same object have completely same representation in the common space. Therefore, this paper proposes robust multi-view common component learning (RMCCL) algorithm, which learns multiple linear transforms to extract the common component of multi-view data from the same instance. To enhance the discriminant ability and robustness of subspace, we introduce binary label matrix technology and serve Cauchy loss as our error measurement. RMCCL can be decomposed into two subproblems by Alternating optimization method, and each subproblem can be optimized by Iteratively Reweight Residuals (IRR) technique. Extensive experiments in both two-view and multi-view datasets demonstrate that the our method outperforms other state-of-the-art approaches.

1 Introduction

In many computer vision applications, one object can be captured by different sensors or observed at diverse viewpoints, e.g., [15]. Thus, one object often exists multiple data representation, usually denoted as multi-view data. Recently, cross-view recognition, which aims to recognize the samples from completely heterogeneous views, becomes more and more significant. However, due to big gap possibly existing between views, the recognition performance will be poor by directly recognizing samples from diverse views.

In order to better handle the cross-view recognition problems, many excellent and effective works have been done. Most of approaches can be attributed to metric learning methods or subspace learning approaches. Owing to the large difference between views, Euclidean distance metric are not applicable. To handle this problem, metric learning methods aim to learn a kind of new similarity measurement manners. Nevertheless, subspace learning try to find a common space shared by multiple views, in which Euclidean distance metric can work. Certainly, there are also many works can be utilized to handle the cross-view recognition problems, such as transfer learning [6, 30]. Because our proposed method belongs to subspace learning, we will introduce this kind of methods in detail. Regarding subspace learning, according to the number of views, this kind of approaches can be grouped into two-view and multi-view approaches.

1.1 Two-view Subspace Learning Approaches

The canonical correlation analysis (CCA) [9] is proposed in 1936, and is the earliest method of cross-view subspace learning. CCA attempts to find two linear transformations, one for each view, such that the correlation coefficient between two views is maximized. To handle information dissipation caused by CCA, Chen et al. [2] propose continuum regression (CR) model. Experiments on cross-modal retrieval demonstrates that CR model can better address the cross-view problems. Heterogeneous face recognition is a specific but important subproblem in the cross-view recognition problems. To better address this subproblem, in [19, 21], PLS is employed to conduct effective feature selection. CCA is only a unsupervised subspace learning approach. To take full advantage of label information, multi-view fisher discriminant analysis (MFDA) [3, 4] effectively utilizes labels information to learn informative projections. Experimental results show that MFDA works better than previous unsupervised methods. For heterogeneous face recognition, Conventional approaches, which bring in a middle conversion stage, are easy to cause performance degradation. In order to better handle this problem, Lin and Tang [17] propose common discriminant feature extraction (CDFE) to simultaneously learn two transformation, which transform two-view samples to common subspace. Certainly, in addition to previously mentioned approaches, there are some excellent works [1, 16, 23, 25].

1.2 Multi-view Subspace Learning Approaches

No doubt that we can use two-view subspace learning approaches to solve multi-view recognition problem by one-versus-one strategy. To efficiently handle this problem, we tend to design multi-view subspace learning methods, which can transform multiple view samples into a latent subspace, simultaneously. In [18, 20], Multi-view Canonical Correlation Analysis (MCCA) is designed as a multiple versions of the CCA. In addition, Sim et al. [22] present Multimodal Discriminant Analysis (MMDA) to decompose samples into independent modes. Recently, In [11], kan et al. propose a novel method named multi-view discriminant analysis (MvDA), which tries to project the multi-view data into a common space, in which within-class samples are close to each other and between-class samples are far away. Kan et al. believe that there exists some relations between different transformations. To exploit these relations, kan et al. [12] develop view-consistency version of MvDA in 2016, which is called MvDA-VC. Besides, low-rank representation (LRR) is good at digging relations between views. Therefore, recently, LRR is also be utilized to handle the cross-view problems, such as SRRS [14], LRCS [5] and LRDE [13].

Fig. 1.

Illustration of our idea. There are five objects in the Figure, and each object has three views. Each number denotes one object, and each color stands for one view. The first three objects from circle class and the last two from rectangle class. As we have seen, three view data are mapped to a unified space by $W_1, W_2, W_3$ projection matrices, in which various data from one object have unique representation. Furthermore, circle class is far from rectangle class and within-class objects are close to each other. (Color figure online)

As mentioned above, one object can generate multiple view data. However, existing subspace learning approaches can only minimize the difference of within-class samples, i.e., the distance of data from the same instance is close in the unified subspace. Intuitively, we can get a better performance for cross-view recognition problem if the heterogeneous data from one object have completely same representation in the latent common subspace. For this reason, this paper learns multiple linear projection matrices, one for each view, to extract the common component of the heterogeneous data. In this paper, to strengthen the robustness to noise, we employ Cauchy loss [27] as the error measurement in our algorithm. Through the above two thoughts, we still can’t learn a discriminative common space. Therefore, we introduce a non-negative label matrix [28] to enlarge the margins between different classes and diminish the discrepancy of samples in the same class. Finally, the proposed method is applied to several cross-view recognition tasks. Experimental results on two heterogeneous face databases demonstrate that the proposed method outperforms previous cross-view recognitions approaches. The fundamental idea of our algorithm is illustrated in Fig. 1.

2 Robust Multi-view Common Component Learning

In this section, we first present the problem formulation, and then bring in label matrix and Cauchy loss to enhance discriminative ability and robustness of algorithm.

2.1 Problem Formulation

Let $\mathcal {X}=\{x_i^1,x_i^2,\ldots ,x_i^n\}_{i=1}^m$ stands for a dataset containing m objects, and each object exists n samples, usually denoted as n views in this area. $x_i^v\in \mathcal {R}^{d_v}$ denotes the vth view of the ith object embedded in $d_v$ dimension space. Generally, there are a large gap between different views, i.e., the between-class samples in a view might have higher similarity than within-class samples from different views. That will lead to poor performance. To better solve this problem, we expect that multiple samples from one object have completely uniform feature representation. Certainly, we can also consider that we extract component features in the process. We denote the linear transformations as $W=\{W_v\}_{v=1}^n$, where $W_v\in \mathcal {R}^{{d_i}\times {d}}$ is the transformation matrix of ith view, and projection results are $\mathcal {Z}=\{z_i\}_{i=1}^m$, where $z_i\in \mathcal {R}^d$ is the mapping result of ith object embedded in d dimension common space.

It is obvious that our objective is to learn a series of view projection matrices $\{W_v\}_{v=1}^n$ to extract common component z. A straightforward method is minimizing the empirical risk $z_i-W_v^\mathrm {T}x_i^v$ in the whole dataset:

$$\begin{aligned} \min _{W}\frac{1}{mn}\sum _{i=1}^m\sum _{v=1}^n\left| \left| z_i-W_v^\mathrm {T}x_i^v\right| \right| _2^2 +c_1\sum _{v=1}^n\left| \left| W_v\right| \right| _F^2+c_2\sum _{i=1}^m\left| \left| z_i\right| \right| _2^2 \end{aligned}$$

(1)

As can be seen in Eq. (1), least square error are used to extract common features of multiple view data, and regularization items are employed to penalize view projection matrices W and generating samples $z_i$, where $c_1>0$, $c_2>0$ are the balance parameters, which are presented to prevent overfitting.

No doubt that Eq. (1) is able to extract the common component of multiple views from one object. However, the constraint of Eq. (1) is too weak, so that it is not enough to obtain a discriminant subspace. To improve the performance of algorithm, like most previous works, we expect to unite within-class samples and separate between-class samples in the common space. Certainly, it is usual that manifold learning methods [7, 8] can be use to improve the discriminant ability of algorithms. However, we adopt a simple but effective methods in this paper. We introduce binary label matrix to enhance the discriminant ability of our algorithm.

2.2 Least Square Regression for Multi-class Classification

Given a dataset $\mathcal {X}=\{(x_i, y_i)\}_{i=1}^m$, where $x_i$ is the ith sample of dataset $\mathcal {X}$, and each input feature $x_i$ corresponds to a target vector $y_i$. As usual, the regression problem can be expressed as

$$\begin{aligned} \min _{W,b}\sum _{i=1}^m\left| \left| W^\mathrm {T}x_i+b-y_i\right| \right| _2^2+\frac{\lambda }{2}\left| \left| W\right| \right| _F^2 \end{aligned}$$

(2)

where $W\in \mathcal {R}^{d\times {p}}$ is weight matrix, $b\in \mathcal {R}^p$ is the bias vector and $\lambda$ is the regularized parameter, which is used to prevent overfitting. In the regression problem, the target $y_i$ in Eq. (2) is continuous. Certainly, by setting target to 0 and 1, Eq. (2) can be extended to two-class classification. Regarding multi-class classification, we can adopt one-versus-one policy to generalize Eq. (2). However, we will have to address multiple subproblems generated by this policy.

In order to use regression model to better handle multi-class classification problem, we introduce label matrix in this paper. Supposing $\mathcal {X}$ is from c classes, and $x_i$ is from class j. Then, for sample $x_i$, its target vector is $y_i = \left[ 0, \ldots , 0, 1, 0,\ldots , 0\right] ^\mathrm {T}$ with only the jth element equal to one, where $y_i\in \mathcal {R}^c$. The target vector constructed by this rule is called as binary label matrix. Then, Eq. (2) can be generalized to multi-class classification easily.

As a matter of fact, numerous works utilize linear regression model to address multi-class classification problem by bringing in binary label matrix [28]. In this paper, we utilize it to restraint common space z learning in Eq. (1). Thus, Eq. (1) can be rewritten as

$$\begin{aligned} \min _{W}\frac{1}{mn}\sum _{i=1}^m\sum _{v=1}^n\left| \left| z_i-W_v^\mathrm {T}x_i^v\right| \right| _2^2 +c_1\sum _{v=1}^n\left| \left| W_v\right| \right| _F^2+c_2\sum _{i=1}^m\left| \left| z_i-y_i\right| \right| _2^2 \end{aligned}$$

(3)

where $y_i$ is the binary label vector of $x_i$. Supposed that the training set is from c classes, we can regard $y_i$ as a point embedded in c dimension space. In this space, within-class samples correspond to the same point and between-class samples are forced to be separated. Therefore, we can obtain a discriminant space by using label matrix. We might as well call Eq. (3) as MCCL-L2 to distinguish what we will propose below.

2.3 Cauchy Loss

Least square loss is generally used as error measurement, as we have done in Eq. (3). However, thoroughly theory studies [10] show that least square estimator is not robust to noise, thus the performance of cross-view recognition will be seriously degraded when the data is not clean. Xu et al. [27] indicates that Cauchy loss is more robust than $L_2$ loss. In order to strength the robustness of algorithm, we employ Cauchy loss to instead of $L_2$ loss, then the objective function can be rewritten as

$$\begin{aligned} \min _{W}\frac{1}{mn}\sum _{i=1}^m\sum _{v=1}^n\log \left( 1+\frac{\left| \left| z_i-W_v^\mathrm {T}x_i^v\right| \right| _2^2}{c^2}\right) +c_1\sum _{v=1}^n\left| \left| W_v\right| \right| _F^2+c_2\sum _{i=1}^m\left| \left| z_i-y_i\right| \right| _2^2 \end{aligned}$$

(4)

where c is a constant. The Eq. (4) jointly models the relationships between each view space $\mathcal {X}^v$ and the common subspace Z by a robust algorithm. It can be expected that we can acquire multiple linear projections by solving this problem with an input of multiple view data. Equation (4) is our method in this paper, named Robust Multi-view Common Component Learning.

3 Optimization

Through alternating optimization method, problem (4) can be decomposed into two subproblems. Inspired by generalized Weiszfeld’s method [24], Xu et al. [27] develop Iteratively Reweight Residuals (IRR) technology. In this paper, we utilize IRR to efficiently optimize the subproblems, respectively.

Firstly, fix view projection matrices $\{W_v\}_{v=1}^n$ and data representation in common subspace $\{z_j\}_{j=1}^m, j\ne {i}$, then Eq. (4) is decomposed into subproblem over $z_i$, which can be written as

$$\begin{aligned} \min _{z_i}\mathcal {J}=\frac{1}{n}\sum _{v=1}^n\log \left( 1+\frac{\left| \left| z_i-W_v^\mathrm {T}x_i^v\right| \right| _2^2}{c^2}\right) +c_2\left| \left| z_i-y_i\right| \right| _2^2 \end{aligned}$$

(5)

setting the gradient of $\mathcal {J}$ with respect to $z_i$ to zero, we have

$$\begin{aligned} \sum _{v=1}^n\frac{z_i-W_v^\mathrm {T}x_i^v}{c^2+\left| \left| z_i-W_v^\mathrm {T}\right| \right| _2^2}+nc_2\left( z_i-y_i\right) =0 \end{aligned}$$

(6)

which can also be rewritten as

$$\begin{aligned} \left( \sum _{v=1}^n\frac{1}{c^2+\left| \left| z_i-W_v^\mathrm {T}x_i^v\right| \right| _2^2}+nc_2\right) z_i =\left( \sum _{v=1}^n\frac{W_v^\mathrm {T}x_i^v}{c^2+\left| \left| z_i-W_v^\mathrm {T}x_i^v\right| \right| _2^2}+nc_2y_i\right) \end{aligned}$$

(7)

where $r^v=z_i-W_v^\mathrm {T}x_i^v$ is referred to as the residual. A weight function is then defined as

$$\begin{aligned} Q=\left[ \frac{1}{c^2+\left| \left| r^1\right| \right| _2^2},\ldots ,\frac{1}{c^2+\left| \left| r^n\right| \right| _2^2}\right] \end{aligned}$$

(8)

Then, put together Eqs. (7) and (8)

$$\begin{aligned} z_i=\left( \left( \sum _{v=1}^nQ_v\right) +nc_2\right) ^{-1}\left( \left( \sum _{v=1}^nW_v^\mathrm {T}x_i^v\right) +nc_2y_i\right) \end{aligned}$$

(9)

It is obvious that Q is the function of $z_i$, we thus utilize IRR technology to iteratively update $z_i$ using Eq. (9) with an initial value until convergence. The procedure is described in Algorithm 1.

By fixing all common subspace data points $\{z_i\}_{i=1}^m$ and view projection matrix $\{W_k\}_{k=1}^n, k\ne {v}$, Eq. (4) is reduced to the following subproblem over projection matrix $W_v$:

$$\begin{aligned} \min _{W_v}\mathcal {J}=\frac{1}{m}\sum _{i=1}^m\log \left( 1+\frac{\left| \left| z_i-W_v^\mathrm {T}x_i^v\right| \right| _2^2}{c^2}\right) +c_1\left| \left| W_v\right| \right| _F^2 \end{aligned}$$

(10)

Setting the derivation of $\mathcal {J}$ with respect to $W_v$ to zero. Finally, the projection matrix $W_v$ can be updated by

$$\begin{aligned} W_v=\left( \left( \sum _{i=1}^mx_i^vQ_i\left( x_i^v\right) ^\mathrm {T}\right) +mc_1\right) ^{-1}\sum _{i=1}^mx_i^vQ_iz_i^\mathrm {T} \end{aligned}$$

(11)

where $Q_i$ is the ith element of Q, and Q can be written as

$$\begin{aligned} Q=\left[ \frac{1}{c^2+\left| \left| r_1\right| \right| _2^2},\ldots ,\frac{1}{c^2+\left| \left| r_m\right| \right| _2^2}\right] \end{aligned}$$

(12)

where $r_i=z_i-W_v^\mathrm {T}x_i^v$. Considering $Q_i$ depends on $W_v$, we thus iteratively update $W_v$ using Eq. (12). The iterative procedure is similar to Algorithm 1.

4 Experiments

In this section, the proposed method RMCCL is evaluated on two heterogeneous face databases, including the Heterogeneous Face Biometrics (HFB) database and the CMU Pose, Illumination, and Expression (PIE) database (CMU PIE). The representative methods CCA [9], MCCA [20], LRCS [5], MvDA [11] and MvDA-VC [12] and MCCL-L2 are brought in for comparison and the traditional single-view methods PCA and LDA serve as baselines. It is worth noting that MCCL-L2 algorithm is proposed by replacing Cauchy loss in Eq. (4) with $L_2$ loss. It can be regard as a middle algorithm of RMCCL. We expect to prove Cauchy loss works better than $L_2$ loss.

Fig. 2.

Selected subjects from the HFB dataset. The first row is visual light images, and the second row is near infrared images

4.1 Experimental Settings

One object can be captured by different sensors, such as visual light camera and near-infrared camera. There are 100 individuals on the HFB dataset, in which each subject has four visual light photos and four near infrared photos. We use 70 individuals to train the RMCCL model, and the rest are utilized to evaluate the performance of our algorithm. Some selected faces from the HFB database can be seen in Fig. 2. Regarding the CMU PIE database, five poses, i.e., C11, C29, C27, C05 and C37, are chosen as the multiple view data, and each subject has four face images under every pose. Some selected faces under five poses can be seen in Fig. 3. 45 people are chosen to train the model, and the rest are employed to evaluate algorithm.

The HFB and the CMU PIE heterogeneous face datasets are cropped according standard protocol and are resized to $32\times 32$ and $64\times 64$, respectively. It is worth noting that, in the HFB or the CMU PIE dataset, there are multiple schemes to divide datasets into training set and testing set. To make the results more convincing, all experiments are repeated ten times by randomly dividing these into two parts. The average results are regarded as the final accuracies (mACC). Furthermore, all methods employ principle component analysis to realize dimensionality reduction, and each approach sets dimension to get the best accuracy.

Fig. 3.

Selected subjects from the CMU PIE dataset. C11, C29, C27, C05 and C37 poses are selected as the views.

4.2 Face Recognition Between Visual Light and Near Infrared Images

In this experiment, we evaluate the performance of RMCCL on the HFB, in which objects are captured by different sensors. The proposed methods RMCCL and MCCL-L2 are compared to classic works including PCA, LDA, CCA [9], MCCA [20], LRCS [5], MvDA [11], MvDA-VC [12]. The experimental results are shown in Table 1.

As can be seen, MvDA-VC is the best competitive method with an high recognition accuracy of 55.88%. However, our proposed method RMCCL achieves a highest recognition rate of $61.46\%$. Note that, because low rank representation methods usually select train set as the over-complete dictionary, LRCS fails to work when dataset is small. Therefore, it obtains a recognition rate of $33.21\%$. It is just a bit better than MCCA. What’s more, our method RMCCL has better performance than MCCL-L2. Obviously, it accords with the theoretical analysis. Through the experimental results on the HFB, it shows that RMCCL outperforms other subspace learning methods in two-view case.

Table 1. The average recognition results $(\%)$ on the HFB dataset

Table 2. The average recognition results $(\%)$ on the CMU PIE dataset

4.3 Face Recognition Across Poses

In the experiments, the CMU PIE dataset is utilized to evaluate face recognition across poses, and pair-wise manner is employed to conduct the experiment. Since each object owns five view data, it generates $5\times 4=20$ recognition accuracies. Then, we utilize the mACC as final evaluation index. The Experimental results on the CMU PIE dataset can be seen in Table 2.

As can be seen, due to unite the within-class samples and separate between-class samples in the latent unified space, the MvDA outperforms PCA, LDA, MCCA on the CMU PIE database. As we have explained in the HFB dataset experiments, since the train set fails to compose a over-complete basic, LRCS performs even worse than MvDA. Through considering the relation of multiple view projections, MvDA-VC further improves performance by $11.7\%$. It is worth noting that, proposed method RMCCL outperforms MvDA-VC with an absolute improvement by $6.1\%$. In addition, RMCCL also performs better than MCCL-L2 with an improvement by $2.8\%$. It demonstrates that Cauchy loss has better robustness than $L_2$ loss. Experimental results show that our method RMCCL performs better than several state-of-the-art algorithms on the CMU PIE dataset. Furthermore, it demonstrates that our approach is also suitable for multi-view data.

5 Conclusion

In this paper, we propose RMCCL algorithm, which learns multiple linear transforms to extract the common component of multi-view data from the same instance. To enhance the discriminant ability and robustness of subspace, we introduce binary label matrix technology and serve Cauchy loss as our error measurement. Experimental results show that our approach outperforms several state-of-the-art algorithms. In future, we’ll further analyze the robust and convergence of algorithm. Certainly, in order to better address various data, we can also bring kernel technology [26, 29] to develop the kernel version of RMCCL.