Robust multiview feature selection via view weighted

Abstract

In recent years, combining the multiple views of data to perform feature selection has been popular. As the different views are the descriptions from different angles of the same data, the abundant information coming from multiple views instead of the single view can be used to improve the performance of identification. In this paper, through the view weighted strategy, we propose a novel robust supervised multiview feature selection method, in which the robust feature selection is performed under the effect of l_2,1-norm. The proposed model has the following advantages. Firstly, different from the commonly used view concatenation that is liable to ignore the physical meaning of features and cause over-fitting, the proposed method divides the original space into several subspaces and performs feature selection in the subspaces, which can reduce the computational complexity. Secondly, the proposed method assigns different weights to views adaptively according to their importance, which shows the complementarity and the specificity of views. Then, the iterative algorithm is given to solve the proposed model, and in each iteration, the original large-scale problem is split into the small-scale subproblems due to the divided original space. The performance of the proposed method is compared with several related state-of-the-art methods on the widely used multiview datasets, and the experimental results demonstrate the effectiveness of the proposed method.

Appendix

1.1 The proof of Theorem 1

According to step 2 in Algorithm 1,

$$ \begin{array}{@{}rcl@{}} W^{t+1}=\min\limits_{W}&& \lambda_{1}\sum\limits_{v=1}^{m}\Vert W_{v}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m}\Vert W_{v}\Vert_{F} \\&&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W_{v}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(21)

Since the following equations hold

$$ \begin{array}{@{}rcl@{}} \|W_{v}\|_{2,1}= tr\left( W_{v}^{\top} D_{1v} W_{v}\right) \end{array} $$

(22)

$$ \begin{array}{@{}rcl@{}} \|W_{v}\|_{F}= tr\left( W_{v}^{\top} D_{2v} W_{v}\right) \end{array} $$

(23)

where D_1v and D_2v are given in (9), (21) can be transformed into

$$ \begin{array}{@{}rcl@{}} W^{t+1}=\min\limits_{W}&&\lambda_{1}\sum\limits_{v=1}^{m}tr\left( W_{v}^{\top} D^{t}_{1v} W_{v}\right)+\lambda_{2}{\sum}_{v=1}^{m}tr\left( W_{v}^{\top} D^{t}_{2v} W_{v}\right) \\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W_{v}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(24)

therefore,

$$ \begin{array}{@{}rcl@{}} &&\lambda_{1}\sum\limits_{v=1}^{m} tr\left( W_{v}^{t+1^{\top}} D^{t}_{1v} W^{t+1}_{v}\right)+\lambda_{2}\sum\limits_{v=1}^{m} tr\left( W_{v}^{t+1^{\top}} D^{t}_{2v} W^{t+1}_{v}\right) \\ &&+ \sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \\ \leq&&\lambda_{1}\sum\limits_{v=1}^{m}tr\left( W_{v}^{t^{\top}} D^{t}_{1v} {W^{t}_{v}}\right)+\lambda_{2}{\sum}_{v=1}^{m}tr\left( W_{v}^{t^{\top}} D^{t}_{2v} {W^{t}_{v}}\right) \\&&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(25)

Substituting D_1v and D_2v with definitions and the following inequalities can be obtained

$$ \begin{array}{@{}rcl@{}} &&\lambda_{1}\sum\limits_{i=1}^{d} \frac{\|\mathbf{w}^{i^{t+1}}\|_{2}^{2}}{2\|\mathbf{w}^{i^{t}}\|_{2}}+\lambda_{2}\sum\limits_{v=1}^{m} \frac{\|W_{v}^{t+1}\|_{F}^{2}}{2\|{W_{v}^{t}}\|_{F}}+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \\ \leq&&\lambda_{1}\sum\limits_{i=1}^{d} \frac{\|\mathbf{w}^{i^{t}}\|_{2}^{2}}{2\|\mathbf{w}^{i^{t}}\|_{2}}+\lambda_{2}\sum\limits_{v=1}^{m} \frac{\|{W_{v}^{t}}\|_{F}^{2}}{2\|{W_{v}^{t}}\|_{F}} +\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(26)

According to Lemma 1. we replace a and b with $\|\mathbf {w}^{i^{t+1}}\|_{2}^{2}$ (or $\|W_{v}^{t+1}\|_{F}^{2}$ ) and $\|\mathbf {w}^{i^{t}}\|_{2}^{2}$ (or $\|{W_{v}^{t}}\|_{F}^{2}$ ), respectively, then the following inequalities can be obtained

$$ \begin{array}{@{}rcl@{}} \|\mathbf{w}^{i^{t+1}}\|_{2}- \frac{\|\mathbf{w}^{i^{t+1}}\|_{2}^{2}}{2\|\mathbf{w}^{i^{t}}\|_{2}}\leq \|\mathbf{w}^{i^{t}}\|_{2} - \frac{\|\mathbf{w}^{i^{t}}\|_{2}^{2}}{2\|\mathbf{w}^{i^{t}}\|_{2}} \end{array} $$

(27)

$$ \begin{array}{@{}rcl@{}} \|W_{v}^{t+1}\|_{F} - \frac{\|W_{v}^{t+1}\|_{F}^{2}}{2\|{W_{v}^{t}}\|_{F}}\leq \|{W_{v}^{t}}\|_{F} - \frac{\|{W_{v}^{t}}\|_{F}^{2}}{2\|{W_{v}^{t}}\|_{F}} \end{array} $$

(28)

Adding (26)–(28) on both sides (note that (27) is repeated for 1 ≤ i ≤ d and (28) is repeated for 1 ≤ v ≤ m), gives

$$ \begin{array}{@{}rcl@{}} && \lambda_{1}\sum\limits_{i=1}^{d} \|\mathbf{w}^{i^{t+1}}\|_{2}+\lambda_{2}\sum\limits_{v=1}^{m} \|W_{v}^{t+1}\|_{F} +\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \\ \leq&& \lambda_{1}\sum\limits_{i=1}^{d} \|\mathbf{w}^{i^{t}}\|_{2}+\lambda_{2}\sum\limits_{v=1}^{m} \|{W_{v}^{t}}\|_{F} +\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(29)

Since

$$ \begin{array}{@{}rcl@{}} \sum\limits_{v=1}^{m}\Vert W_{v}\Vert_{2,1}=\sum\limits_{i=1}^{d} \|\mathbf{w}^{i}\|_{2} \end{array} $$

(30)

Equation (29) can be transformed as follows

$$ \begin{array}{@{}rcl@{}} &&\lambda_{1}\sum\limits_{v=1}^{m}\Vert W^{t+1}_{v}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|W_{v}^{t+1}\|_{F} +\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \\ \leq&&\lambda_{1}\sum\limits_{v=1}^{m}\Vert {W^{t}_{v}}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|{W_{v}^{t}}\|_{F}+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(31)

According to the step 4 in Algorithm 1,

$$ \begin{array}{@{}rcl@{}} E_{v}^{t+1}=\min\limits_{E_{v}}\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert E_{v}\Vert_{2,1}+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E_{v}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(32)

thus,

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert E^{t+1}_{v}\Vert_{2,1}+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}}\\ &&\leq\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert {E^{t}_{v}}\Vert_{2,1}+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(33)

Combining Eqs. (31) and (33), the following inequality can be obtained

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert E^{t+1}_{v}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert W^{t+1}_{v}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|W_{v}^{t+1}\|_{F} \\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \\ \leq&&\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert {E^{t}_{v}}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert {W^{t}_{v}}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|{W_{v}^{t}}\|_{F}\\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(34)

According to the step 5 in Algorithm 1,

$$ \begin{array}{@{}rcl@{}} \theta_{v}^{t+1}=\min\limits_{\theta_{v}}\sum\limits_{v=1}^{m} (\theta_{v})^{p} \Vert E^{t+1}_{v}\Vert_{2,1} \end{array} $$

(35)

thus,

$$ \begin{array}{@{}rcl@{}} \sum\limits_{v=1}^{m} (\theta_{v}^{t+1})^{p} \Vert E^{t+1}_{v}\Vert_{2,1}\leq\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert E^{t+1}_{v}\Vert_{2,1} \end{array} $$

(36)

Combining (34) and (36), the following inequality can be obtained

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} (\theta_{v}^{t+1})^{p} \Vert E^{t+1}_{v}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert W^{t+1}_{v}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|W_{v}^{t+1}\|_{F} \\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \\ \leq&&\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert {E^{t}_{v}}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert {W^{t}_{v}}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|{W_{v}^{t}}\|_{F}\\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(37)

According to the step 6 in Algorithm 1,

$$ \begin{array}{@{}rcl@{}} {\Lambda}_{v}^{t+1}=\min\limits_{{\Lambda}_{v}}\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{\Lambda}_{v}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t}}{2}\Vert\frac{1}{\mu^{t}}{\Lambda}_{v}{\Vert_{F}^{2}} \end{array} $$

(38)

thus,

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t}}{2}\Vert\frac{1}{\mu^{t}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}} \\ \leq&&\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t}}{2}\Vert\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(39)

Combining (37) and (39), the following inequality can be obtained

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} (\theta_{v}^{t+1})^{p} \Vert E^{t+1}_{v}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert W^{t+1}_{v}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|W_{v}^{t+1}\|_{F} \\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t}}{2}\Vert\frac{1}{\mu^{t}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}} \\ \leq&&\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert {E^{t}_{v}}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert {W^{t}_{v}}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|{W_{v}^{t}}\|_{F}\\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t}}{2}\Vert\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(40)

According to the step 7 in Algorithm 1,

$$ \begin{array}{@{}rcl@{}} \mu^{t+1}=\min\limits_{\mu}\sum\limits_{v=1}^{m} \frac{\mu}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu}{2}\Vert\frac{1}{\mu}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}} \end{array} $$

(41)

thus,

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} \frac{\mu^{t+1}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t+1}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t+1}}{2}\Vert\frac{1}{\mu^{t+1}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}} \\ \leq&&{\sum}_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t}}{2}\Vert\frac{1}{\mu^{t}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}} \end{array} $$

(42)

Combining (40) and (42), the following inequality can be obtained

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} (\theta_{v}^{t+1})^{p} \Vert E^{t+1}_{v}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert W^{t+1}_{v}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|W_{v}^{t+1}\|_{F} \\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t+1}}{2} \Vert X_{v}^{\top} W^{t+1}_{v}-Y-E^{t+1}_{v}+\frac{1}{\mu^{t+1}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t+1}}{2}\Vert\frac{1}{\mu^{t+1}}{\Lambda}_{v}^{t+1}{\Vert_{F}^{2}} \\ \leq&&\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert {E^{t}_{v}}\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert {W^{t}_{v}}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|{W_{v}^{t}}\|_{F}\\ &&+\sum\limits_{v=1}^{m} \frac{\mu^{t}}{2} \Vert X_{v}^{\top} {W^{t}_{v}}-Y-{E^{t}_{v}}+\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}}-\sum\limits_{v=1}^{m}\frac{\mu^{t}}{2}\Vert\frac{1}{\mu^{t}}{{\Lambda}_{v}^{t}}{\Vert_{F}^{2}} \end{array} $$

(43)

Since $X_{v}^{\top } W_{v}-Y$ is replaced with E_v before, (43) can be transformed as follows

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{v=1}^{m} (\theta_{v}^{t+1})^{p} \Vert X_{v}^{{t+1}\top} W_{v}^{t+1}-Y\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert W^{t+1}_{v}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|W_{v}^{t+1}\|_{F} \\ \leq&&\sum\limits_{v=1}^{m} ({\theta_{v}^{t}})^{p} \Vert X_{v}^{{t}\top} {W_{v}^{t}}-Y\Vert_{2,1}+\lambda_{1}\sum\limits_{v=1}^{m}\Vert {W^{t}_{v}}\Vert_{2,1}+\lambda_{2}\sum\limits_{v=1}^{m} \|{W_{v}^{t}}\|_{F} \end{array} $$

(44)

Thus,

$$ \begin{array}{@{}rcl@{}} Obj(t+1)\leq Obj(t) \end{array} $$

(45)

Equation (45) indicates that the value of the objective function (2) is decreased in each iteration of the Algorithm 1. And beacuse (2) is greater than zero, Theorem 1 is proven.