Bounding the difference between RankRC and RankSVM and application to multi-level rare class kernel ranking

Abstract

Rapid explosion in data accumulation has yielded large scale data mining problems, many of which have intrinsically unbalanced or rare class distributions. Standard classification algorithms, which focus on overall classification accuracy, often perform poorly in these cases. Recently, Tayal et al. (IEEE Trans Knowl Data Eng 27(12):3347–3359, 2015) proposed a kernel method called RankRC for large-scale unbalanced learning. RankRC uses a ranking loss to overcome biases inherent in standard classification based loss functions, while achieving computational efficiency by enforcing a rare class hypothesis representation. In this paper we establish a theoretical bound for RankRC by establishing equivalence between instantiating a hypothesis using a subset of training points and instantiating a hypothesis using the full training set but with the feature mapping equal to the orthogonal projection of the original mapping. This bound suggests that it is optimal to select points from the rare class first when choosing the subset of data points for a hypothesis representation. In addition, we show that for an arbitrary loss function, the Nyström kernel matrix approximation is equivalent to instantiating a hypothesis using a subset of data points. Consequently, a theoretical bound for the Nyström kernel SVM can be established based on the perturbation analysis of the orthogonal projection in the feature mapping. This generally leads to a tighter bound in comparison to perturbation analysis based on kernel matrix approximation. To further illustrate computational effectiveness of RankRC, we apply a multi-level rare class kernel ranking method to the Heritage Health Provider Network’s health prize competition problem and compare the performance of RankRC to other existing methods.

Appendix A: Proof Sketch for Theorem 5

Proof

Assume that $\mathbf {w}^*$ and $\mathbf {w}_{\mathscr {R}}^*$ are minimizers of (28) and (30), respectively, with total loss for multi-level RankSVM given by:

$$\begin{aligned} R_\phi (\mathbf {w}) = \frac{1}{\sum _{r < s} m_r m_s} \sum _{r=1}^R \sum _{ \{i : y_i = r\} } \sum _{ \{j : y_i > y_j \}} \ell _h \left( \mathbf {w}^T\phi (\mathbf {x}_i) - \mathbf {w}^T\phi (\mathbf {x}_j) \right) \end{aligned}$$

(A.1)

and

$$\begin{aligned} R_{\phi _{\mathscr {R}}}(\mathbf {w}) = \frac{1}{\sum _{r < s} m_r m_s} \sum _{r=1}^R \sum _{ \{i : y_i = r\} } \sum _{ \{j : y_i > y_j \}} \ell _h \left( \mathbf {w}^T\phi _{\mathscr {R}}(\mathbf {x}_i) - \mathbf {w}^T\phi _{\mathscr {R}}(\mathbf {x}_j) \right) . \end{aligned}$$

(A.2)

Let $\Delta \mathbf {w}= \mathbf {w}_{\mathscr {R}}^* - \mathbf {w}^*$.

A convex function g satisfies

$$\begin{aligned} g(\mathbf {u}+ t(\mathbf {v}-\mathbf {u})) - g(\mathbf {u}) \le t ( g(\mathbf {v}) - g(\mathbf {u}) ) \end{aligned}$$

for all $\mathbf {u}, \mathbf {v}, t \in [0,1]$. Since $\ell _h$ is convex, $R_\phi$ and $R_{\phi _{\mathscr {R}}}$ are convex. Then

$$\begin{aligned} R_{\phi }( \mathbf {w}^* + t \Delta \mathbf {w}) - R_{\phi }( \mathbf {w}^*)&\le t ( R_{\phi }(\mathbf {w}_{\mathscr {R}}^*) - R_{\phi }( \mathbf {w}^*) )\end{aligned}$$

(A.3)

$$\begin{aligned} \text {and} \quad R_{\phi _{\mathscr {R}}}( \mathbf {w}_{\mathscr {R}}^* - t \Delta \mathbf {w}) - R_{\phi _{\mathscr {R}}}( \mathbf {w}_{\mathscr {R}}^*)&\le t ( R_{\phi _{\mathscr {R}}}( \mathbf {w}^*) - R_{\phi _{\mathscr {R}}}(\mathbf {w}_{\mathscr {R}}^*)), \end{aligned}$$

(A.4)

for all $t \in [0,1]$.

Since $\mathbf {w}^*$ and $\mathbf {w}_{\mathscr {R}}^*$ are minimizers of $F_\phi$ and $F_{\phi _{\mathscr {R}}}$, for any $t \in [0,1]$, we have

$$\begin{aligned} F_{\phi }( \mathbf {w}^*)&\le F_{\phi }(\mathbf {w}^* + t \Delta \mathbf {w}) \end{aligned}$$

(A.5)

$$\begin{aligned} \text {and} \quad F_{\phi _{\mathscr {R}}}( \mathbf {w}_{\mathscr {R}}^*)&\le F_{\phi _{\mathscr {R}}}(\mathbf {w}_{\mathscr {R}}^* - t \Delta \mathbf {w}). \end{aligned}$$

(A.6)

Summing (A.5) and (A.6), using $F_\phi (\mathbf {w}) = R_\phi (\mathbf {w}) + \frac{\lambda }{2} \Vert \mathbf {w}\Vert _2^2$ and the identity

$$\begin{aligned} \left( \Vert \mathbf {w}^*\Vert ^2 - \Vert \mathbf {w}^* + t \Delta \mathbf {w}\Vert ^2 \right) + \left( \Vert \mathbf {w}_{\mathscr {R}}^*\Vert ^2 - \Vert \mathbf {w}_{\mathscr {R}}^* - t \Delta \mathbf {w}\Vert ^2 \right) = 2t(1-t)\Vert \Delta \mathbf {w}\Vert ^2, \end{aligned}$$

we obtain

$$\begin{aligned} \lambda t(1-t) \Vert \Delta \mathbf {w}\Vert ^2 \le \left( R_{\phi }( \mathbf {w}^* + t \Delta \mathbf {w}) - R_{\phi }( \mathbf {w}^*) \right) + \left( R_{\phi _{\mathscr {R}}}(\mathbf {w}_{\mathscr {R}}^* - t \Delta \mathbf {w}) - R_{\phi _{\mathscr {R}}}( \mathbf {w}_{\mathscr {R}}^*) \right) \end{aligned}$$

(A.7)

Substituting (A.3) and (A.4) into (A.7), dividing by $\lambda t$, and taking the limit $t \rightarrow 0$ gives

$$\begin{aligned}&\Vert \Delta \mathbf {w}\Vert ^2\\&\quad \le \frac{1}{\lambda } \left( R_{\phi }(\mathbf {w}_{\mathscr {R}}^*) - R_{\phi _{\mathscr {R}}}( \mathbf {w}_{\mathscr {R}}^*) + R_{\phi _{\mathscr {R}}}( \mathbf {w}^*) - R_{\phi }( \mathbf {w}^*) \right) \\&\quad = \frac{1}{\lambda \sum _{r < s} m_r m_s} \sum _{r=1}^R \sum _{ \{i : y_i = r\} } \sum _{ \{j : y_i > y_j \}} \Big [ \ell _h \left( (\mathbf {w}_{\mathscr {R}}^*)^T \phi (\mathbf {x}_i) - (\mathbf {w}_{\mathscr {R}}^*)^T \phi (\mathbf {x}_j) \right) \\&\qquad -\,\ell _h \left( (\mathbf {w}_{\mathscr {R}}^*)^T \phi _{\mathscr {R}}(\mathbf {x}_i) - (\mathbf {w}_{\mathscr {R}}^*)^T \phi _{\mathscr {R}}(\mathbf {x}_j) \right) + \ell _h \left( (\mathbf {w}^*)^T \phi _{\mathscr {R}}(\mathbf {x}_i) - (\mathbf {w}^*)^T \phi _{\mathscr {R}}(\mathbf {x}_j) \right) \\&\qquad - \,\ell _h \left( (\mathbf {w}^*)^T \phi (\mathbf {x}_i) - (\mathbf {w}^*)^T \phi (\mathbf {x}_j) \right) \Big ] \\ \end{aligned}$$

where the last inequality uses the multi-level RankSVM definitions of $R_\phi$ and $R_{\phi _{\mathscr {R}}}$. Since $\ell _h(\cdot )$ is 1-Lipschitz, we obtain

(A.8)

where the last equality is obtained by factoring over common index sets $\{i : y_i = r\}$.

From $\phi (\mathbf {x}) = \phi _{\mathscr {R}}(\mathbf {x}) + \phi _{\mathscr {R}}^\perp (\mathbf {x})$, with $\phi _{\mathscr {R}}(\mathbf {x}) \in {\mathscr {S}}_{\mathscr {R}}$ and $\phi _{\mathscr {R}}^\perp (\mathbf {x})$ is in the space orthogonal to ${\mathscr {S}}_{\mathscr {R}}$, we have, for $i=1,\ldots ,m$,

$$\begin{aligned} \Vert \phi (\mathbf {x}_i) - \phi _{\mathscr {R}}(\mathbf {x}_i) \Vert = \Vert \phi _{\mathscr {R}}^\perp (\mathbf {x}_i) \Vert \le {\left\{ \begin{array}{ll} \Vert \phi (\mathbf {x}_i) \Vert = \sqrt{ k(\mathbf {x}_i,\mathbf {x}_i) } \le \sqrt{\kappa } , &{} \text {if}\ i \not \in {\mathscr {R}} \\ 0, &{} \text {if}\ i \in {\mathscr {R}}. \end{array}\right. } \end{aligned}$$

(A.9)

In addition, recall that RankSVM is equivalent to a 1-class SVM on an enlarged dataset with the set of points $\mathscr {P} = \{ \phi (\mathbf {x}_i) - \phi (\mathbf {x}_j) : y_i > y_j, i,j=1\ldots ,m\}$. Therefore $\mathbf {w}$ can be expressed in terms of the dual variables $0 \le \alpha _{ij}^* \le C$ of an SVM problem trained on $\mathscr {P}$ with $C = \frac{1}{\lambda m_+ m_-}$, as follows,

$$\begin{aligned} \mathbf {w}^*&= \sum _{ \{ i,j: y_i > y_j\}} \alpha _{ij}^* ( \phi (\mathbf {x}_i) - \phi (\mathbf {x}_j) ) = \sum _{ \{i : y_i = +1 \} } \sum _{ \{ j : y_j = -1 \}} \alpha _{ij}^* ( \phi (\mathbf {x}_i) - \phi (\mathbf {x}_j) ) \\&= \sum _{ \{i : y_i = +1 \} } \phi (\mathbf {x}_i) \left( \sum _{ \{ j : y_j = -1 \}} \alpha _{ij}^* \right) - \sum _{ \{ j : y_j = -1 \}} \phi (\mathbf {x}_j) \left( \sum _{ \{i : y_i = +1 \} } \alpha _{ij}^* \right) . \end{aligned}$$

Since $\Vert \phi (\mathbf {x}) \Vert \le \sqrt{\kappa }$ and $C = \frac{1}{\lambda m_+ m_-}$, we get $\Vert \mathbf {w}^*\Vert \le \sqrt{\kappa } C m_- m_+ + \sqrt{\kappa } C m_+ m_- = \frac{2\sqrt{\kappa }}{\lambda }$. Similarly, $\Vert \phi _{\mathscr {R}}(\mathbf {x}) \Vert \le \Vert \phi (\mathbf {x}) \Vert \le \sqrt{\kappa }$ and $\Vert \mathbf {w}_{\mathscr {R}}^* \Vert \le \frac{2\sqrt{\kappa }}{\lambda }$. Together with (A.9), we can then bound (A.8) by

$$\begin{aligned} \Vert \Delta \mathbf {w}\Vert ^2&\le \frac{ 4 \kappa }{\lambda ^2 M } \sum _{r=0}^R \Bigg ( (m_0 + m_1 + \cdots + m_{r-1} + m_{r+1} + \cdots + m_R ) \sum _{ \{i : y_i = r\} } \mathbb {I}[i \not \in {\mathscr {R}}] \Bigg ), \end{aligned}$$

where $M = \sum _{r < s} m_r m_s$.

Therefore, we obtain

$$\begin{aligned} | f_{\phi _{\mathscr {R}}}(\mathbf {x}) - f_\phi (\mathbf {x}) |&= | \mathbf {w}_{\mathscr {R}}^T \phi _{\mathscr {R}}(\mathbf {x}) - \mathbf {w}^T \phi (\mathbf {x}) | \\&= | \mathbf {w}_{\mathscr {R}}^T \left( \phi (\mathbf {x}) - \phi _{\mathscr {R}}^\perp (\mathbf {x}) \right) - \mathbf {w}^T \phi (\mathbf {x}) | \\&= | \Delta \mathbf {w}^T \phi (\mathbf {x}) - \mathbf {w}_{\mathscr {R}}^T \phi _{\mathscr {R}}^\perp (\mathbf {x}) | \\&= | \Delta \mathbf {w}^T \phi (\mathbf {x}) | \\&\le \Vert \Delta \mathbf {w}\Vert \Vert \phi (\mathbf {x}) \Vert \\&\le \frac{ 2 \kappa }{\lambda \sqrt{M}} \Bigg [ \sum _{r=0}^R \Bigg ( (m_0 + m_1 + \cdots + m_{r-1} + m_{r+1} + \cdots + m_R )\\&\quad \sum _{ \{i : y_i = r\} } \mathbb {I}[i \not \in {\mathscr {R}}] \Bigg )\Bigg ]^{\frac{1}{2}}. \end{aligned}$$

This completes the proof. $\square$