Noname manuscript No.

(will be inserted by the editor)

Speed Up Kernel Discriminant Analysis

Deng Cai · Xiaofei He · Jiawei Han

Received: 13 December 2009 /Revised: 7 April 2010/ Accepted: 16 April 2010

Abstract Linear Discriminant Analysis (LDA) has been fully utilize the computational results of the existing
a popular method for dimensionality reduction which training samples. Moreover, it is easy to produce sparse
preserves class separability. The projection vectors are projections (Sparse KDA) with a L1 -norm regularizer.
commonly obtained by maximizing the between class Extensive experiments on spoken letter, handwritten
covariance and simultaneously minimizing the within digit image and face image data demonstrate the effec-
class covariance. LDA can be performed either in the tiveness and efficiency of the proposed algorithm.
original input space or in the reproducing kernel Hilbert
space (RKHS) into which data points are mapped, which Keywords Kernel Discriminant Analysis · Regres-
leads to Kernel Discriminant Analysis (KDA). When sion · Subspace Learning · Dimensionality Reduction
the data are highly nonlinear distributed, KDA can
achieve better performance than LDA. However, com-
puting the projective functions in KDA involves eigen- 1 Introduction
decomposition of kernel matrix, which is very expen-
sive when a large number of training samples exist. Dimensionality reduction has been a key problem in
In this paper, we present a new algorithm for kernel many fields of information processing, such as data min-
discriminant analysis, called Spectral Regression Ker- ing, information retrieval, and pattern recognition. When
nel Discriminant Analysis (SRKDA). By using spectral data is represented as points in a high-dimensional space,
graph analysis, SRKDA casts discriminant analysis into one is often confronted with tasks like nearest neighbor
a regression framework which facilitates both efficient search. Many methods have been proposed to index the
computation and the use of regularization techniques. data for fast query response, such as K-D tree, R tree,
Specifically, SRKDA only needs to solve a set of regu- R* tree, etc [16]. However, these methods can only op-
larized regression problems and there is no eigenvector erate with small dimensionality, typically less than 100.
computation involved, which is a huge save of computa- The effectiveness and efficiency of these methods drop
tional cost. The new formulation makes it very easy to exponentially as the dimensionality increases, which is
develop incremental version of the algorithm which can commonly referred to as the “curse of dimensionality”.
During the last decade, with the advances in com-
Deng Cai · Xiaofei He
puter technologies and the advent of the World Wide
State Key Lab of CAD&CG, Web, there has been an explosion in the amount and
College of Computer Science, complexity of digital data being generated, stored, ana-
Zhejiang University lyzed, and accessed. Much of this information is multi-
Tel.: +86-571-88206681
Fax: +86-571-88206680
media in nature, including text, image, and video data.
E-mail: {dengcai,xiaofeihe}@cad.zju.edu.cn The multimedia data are typically of very high dimen-
Jiawei Han
sionality, ranging from several thousands to several hun-
Department of Computer Science, dreds of thousands. Learning with such high dimension-
University of Illinois at Urbana Champaign ality in many cases is almost infeasible. Thus, learnabil-
E-mail: hanj@cs.uiuc.edu ity necessitates dimensionality reduction [8, 30, 33, 34].
2

Once the high-dimensional data is mapped into lower- these similar techniques can be applied on non-linear
dimensional space, conventional indexing schemes can dimensionality reduction algorithms which use kernel
then be applied [20, 28, 29]. techniques.
One of the most popular dimensionality reduction In this paper, we propose a new algorithm for kernel
algorithms might be Linear Discriminant Analysis (LDA). discriminant analysis, called Spectral Regression Kernel
LDA is a supervised method that has proved successful Discriminant Analysis (SRKDA). Our analysis essen-
on classification problems [9, 15]. The projection vec- tially follows our previous idea for speeding up LDA
tors are commonly obtained by maximizing the be- [6]. By using spectral graph analysis, SRKDA casts dis-
tween class covariance and simultaneously minimizing criminant analysis into a regression framework which
the within class covariance. The classical LDA is a lin- facilitates both efficient computation and the use of reg-
ear method and fails for nonlinear problems. To deal ularization techniques. Specifically, SRKDA only needs
with this limitation, nonlinear extensions of LDA through to solve a set of regularized regression problems and
“kernel trick” have been proposed. The main idea of there is no eigenvector computation involved, which is
kernel-based methods is to map the input data to a a huge save of computational cost. Moreover, the new
feature space through a nonlinear mapping, where the formulation makes it very easy to develop incremen-
inner products in the feature space can be computed by tal version of the algorithm which can fully utilize the
a kernel function without knowing the nonlinear map- previous computational results on the existing training
ping explicitly [27]. Kernel Fisher Discriminant Analy- samples.
sis (KFD) in [22] and Generalized Discriminant Anal- The points below highlight the contributions of this
ysis (GDA) in [1] are two independently developed ap- paper:
proaches for kernel-based nonlinear extensions of LDA.
They are essentially equivalent. To avoid confusion, we – KDA in the binary-class case has been shown to be
will refer this approach as Kernel Discriminant Analysis equivalent to regularized kernel regression with the
(KDA) hereafter. class label as the output [27]. Our paper extends
When solving the optimization problem of KDA, this relation to multi-class case.
we need to handle the possible singularity problem of – We provides a new formulation of KDA optimiza-
the total scatter matrix. There are two approaches try- tion problem. With this new formulation, the KDA
ing to address this issue either by using regularization optimization problem can be efficiently solved by
techniques [22] or by applying singular value decom- avoiding the eigen-decomposition of the kernel ma-
position [1, 26][35]. Both of these two approaches for trix. Theoretical analysis shows that the new ap-
solving optimization problem of KDA involve the eigen- proach can achieve 27-times speedup over the ordi-
decomposition of the kernel matrix which is computa- nary KDA approaches.
tionally expensive. Moreover, due to the difficulty of de- – Moreover, SRKDA can be naturally performed in
signing an incremental solution for the eigen-decomposition the incremental manner. The computational results
on the kernel matrix, there has been little work on de- on the existing training samples can be fully uti-
signing incremental KDA algorithms that can efficiently lized when new training samples are injected into
incorporate new data examples as they become avail- the system. Theoretical analysis shows that SRKDA
able. in the incremental mode has only quadratic-time
In [23], S. Mika et al. made a first attempt to speed complexity, which is a huge improvement comparing
up KDA through a greedy approximation technique. to the cubic-time complexity of the ordinary KDA
However, their algorithm was developed to handle the approaches.
binary classification problem. For a multi-class prob- – Since SRKDA uses regression as a building block,
lem, the authors suggested the one against the rest various kinds of regularization techniques can be
scheme by considering all two-class problems. Recent easily incorporated (e.g., L1 -norm regularizer to pro-
studies [4–7, 9] show that various linear dimensionality duce sparse projections). Our approach provides a
reduction algorithms can be formulated as regression huge possibility to develop new variations of kernel
problems and thus have efficient computational solu- discriminant analysis.
tions. Particularly, our previous work [6] has demon- – A short version of this work has been published in
strated that LDA can be formulated as a regression ICDM [3]. In this journal version, we provide two
problem and be efficiently solved. The similar idea has new sections on theoretical analysis (Section 3.1)
been applied to unsupervised dimensionality reduction and sparse KDA (Section 5). Moreover, we have
algorithms [7] and semi-supervised dimensionality re- added significant amount of the experimental re-
duction algorithms [5]. However, it is not clear that how sults.
 3

The remainder of the paper is organized as follows. Table 1 Notations

In Section 2, we provide a brief review of LDA and Notations Descriptions
KDA, plus a detailed computational analysis of KDA. m the number of total training data points
Section 3 introduces our proposed Spectral Regression n the number of features
c the number of classes
Kernel Discriminant Analysis algorithm. The incremen- mk the number of data points in k-th class
tal version of SRKDA is introduced in Section 4 and the xi the i-th data point
(k)
extensive experimental results are presented in Section xi the i-th data point in the k-th class
5. Finally, we provide some concluding remarks in Sec- µ the total sample mean vector
tion 6. µ (k) the mean vector of the k-th class
X the data matrix
Sb the between-class scatter matrix
Sw the within-class scatter matrix
2 A Brief Review of LDA and KDA St the total scatter matrix
α the transformation vector
Linear Discriminant Analysis (LDA) seeks directions on
which the data points of different classes are far from
For a proper chosen φ, an inner product h, i can be
each other while requiring data points of the same class
defined on F which makes for a so-called reproducing
to be close to each other [15]. Suppose we have a set of
kernel Hilbert space (RKHS). More specifically,
m samples x1 , x2 , · · · , xm ∈ Rn , belonging to c classes.
The objective function of LDA is as follows: hφ(x), φ(y)i = K(x, y)

aT S b a holds where K(., .) is a positive semi-definite kernel func-

aopt = arg max , (1) tion. Several popular kernel functions are: Gaussian ker-
aT S w a
nel K(x, y) = exp(−kx − yk2 /2σ 2 ); polynomial ker-
c
X nel K(x, y) = (1 + xT y)d ; Sigmoid kernel K(x, y) =
Sb = µ(k) − µ )(µ
mk (µ µ(k) − µ )T ,
tanh(xT y + α).
k=1
Let Sbφ , Sw
φ
and Stφ denote the between-class, within-
c mk
!
X X (k) (k)
class and total scatter matrices in the feature space,
(k) (k) T
Sw = (xi −µ )(xi −µ ) , respectively. We have
k=1 i=1
c
X (k) (k)
where µ is the global centroid, mk is the number of Sbφ = µ φ − µ φ )T ,
µφ − µ φ )(µ
mk (µ
samples in the k-th class, µ (k) is the centroid of the k- k=1
(k)
th class, and xi is the i-th sample in the k-th class. mk 
!
c
X X  T
We call Sw the within-class scatter matrix and Sb the φ
Sw =
(k)
φ(xi )
(k)
− µφ
(k)
φ(xi )
(k)
− µφ ,
between-class scatter matrix. i=1
Pm k=1
Define the total scatter matrix St = i=1 (xi − m 
X  T
µ )(xi − µ )T , we have St = Sb + Sw [15]. The objec- Stφ = φ(xi ) − µ φ φ(xi ) − µ φ ,
tive function of LDA in Eqn. (1) is equivalent to i=1
(k)
aT S b a where µ φ and µ φ are the centroids of the k-th class and
aopt = arg max T . (2) the global centroid, respectively, in the feature space.
a St a
Let ν denote the projective function in the feature
The optimal a’s are the eigenvectors corresponding to space, the corresponding objective function (2) in the
the non-zero eigenvalue of eigen-problem: feature space is

Sb a = λSt a. (3) ν T Sbφν

ν opt = arg max , (4)
ν T Stφν
Since the rank of Sb is bounded by c − 1, there are
at most c − 1 eigenvectors corresponding to non-zero which can be solved by the eigen-problem:
eigenvalues [15]. Sbφν = λStφν .
For the sake of clarity, we provide in a summary
for notations usage in Table 1. To extend LDA to the Because the eigenvectors are linear combinations of φ(xi )
nonlinear case, we consider the problem in a feature [1][27], there exist coefficients αi such that
space F induced by some nonlinear mapping m
X
ν= αi φ(xi ).
φ : Rn → F i=1
4

Let α = [α1 , · · · , αm ]T , it can be proved [1] that Eqn. Σr = diag(σ1 , · · · , σr ) is the diagonal matrix of nonzero
(4) is equivalent to: eigenvalues and Ur is the first r columns of U . Thus Σr−1
exists and UrT Ur = I, where I is the identity matrix.
αT KW Kα
α
α opt = arg max , (5) Substituting K in Eqn. (5), we get
T
α KKαα

T

and the corresponding eigen-problem is: Σr UrT α UrT W Ur Σr UrT α
α opt = arg max
T
.
α = λKKα
KW Kα α. (6) Σr UrT α UrT Ur Σr UrT α

where K is the kernel matrix (Kij = K(xi , xj )) and W We proceed to variable modification using β = Σr UrT α
is defined as: and get:

 1/mk , if xk and xj both belong to
Wij = the k-th class; (7) β T UrT W Urβ
 β opt = arg max ,
0, otherwise. βTβ

Each eigenvector α gives a projective function ν in Thus, the optimal β ’s are the leading eigenvectors of
the feature space. For a data example x, we have matrix UrT W Ur . Once β ’s are calculated, α can be com-
m
puted as α = Ur Σr−1β .
X
hνν , φ(x)i = αi hφ(xi ), φ(x)i The second method is using the idea of regulariza-
i=1 tion, by adding constant values to the diagonal elements
Xm of KK, as KK + γI, for γ > 0. It is easy to see that
= αi K(xi , x) KK + γI is nonsingular. This method is used in [22].
i=1 By noticing that
T
= α K(:, x)
. KK + γI = U ΣU T U ΣU T + γI = U (Σ 2 + γI)U T ,
where K(:, x) = [K(x1 , x), · · · , K(xm , x)]T . Let {α α1 , · · · , α c−1 }
be the c − 1 eigenvectors of the eigen-problem in Eqn. we define Σ e = (Σ 2 + γI)1/2 , the objective function of
(6) with respect to the non-zero eigenvalues. The trans- regularized KDA can be written as:
formation matrix Θ = [α α1 , · · · , α c−1 ] is an m × (c − 1)
matrix and a data sample x can be embedded into c − 1 α T KW Kα α
dimensional subspace by max
αT (KK + γI)α α
x → z = ΘT K(:, x). α U ΣU W U ΣU T α
T T
= max
αT U Σ e ΣU
e Tα
The above approach extends LDA into RKHS by βT Σe −1 ΣU T W U Σ Σ e −1β
using “kernel trick” is independently developed by Mika = max T
β β
et al . [22] and Baudat et al . [1]. This algorithm was
named as Kernel Fisher Discriminant (KFD) in [22] and e T α . The optimal β ’s are the leading
where β = ΣU
Generalized Discriminant Analysis (GDA) in [1]. e −1 ΣU T W U Σ Σ
e −1 . With this
eigenvectors of matrix Σ
formulation, the above two methods can be computed
2.1 Computational Analysis of KDA in exactly the same way.
To reduce the computation in calculating β , we shall
To get a stable solution of the eigen-problem in Eqn. exploit the special structure of W . Without loss of gen-
(6), the matrix KK is required to be non-singular [17]. erality, we assume that the data points are ordered ac-
When K is singular, there are two methods to solve this cording to their labels. It is easy to check that the ma-
problem. The first method is by using eigen-decomposition trix W has a block-diagonal structure
of K, which was proposed in [1].  (1) 
Suppose the rank of K is r(r ≤ m) and the eigen- W 0 ··· 0
 0 W (2) · · · 0 
decomposition of K is as follows:  
W = . .. . . ..  (8)
 .
. . . . 
K = U ΣU T = Ur Σr UrT
0 0 · · · W (c)
where Σ = diag(σ1 , · · · , σm ) is the diagonal matrix
of sorted eigenvalues (σ1 ≥ · · · ≥ σm ≥ 0) and U is where {W (k) }ck=1 is an mk × mk matrix with all the
the matrix of normalized eigenvectors associated to Σ. elements equal to 1/mk .
 5

(1) (c)
We partition the m×r matrix Ur as [Ur , · · · , Ur ]T , 3 Efficient KDA via Spectral Regression
(k) (k)
where Ur ∈ Rr×mk . Let vi be the i-th column vector
(k)
of Ur , we have: In order to solve the KDA eigen-problem in Eqn. (6)
efficiently, we use the following theorem:
c
X Theorem 1 Let y be the eigenvector of eigen-problem
UrT W Ur = Ur(k) W (k) (Ur(k) )T
k=1 W y = λy (10)
c mk mk
!
X 1 X (k)
X (k)
= vi (vi )T α = y, then α is the eigenvector
with eigenvalue λ. If Kα
mk
k=1 i=1 i=1 of eigen-problem in Eqn. (6) with the same eigenvalue
Xc
λ.
= mk v̄(k) (v̄(k) )T
k=1 Proof We have W y = λy. At the left side of Eqn. (6),
=HH T α by y, we have
replace Kα
√ √  α = KW y = Kλy = λKy = λKKα
KW Kα α
where H = m1 v̄(1) , · · · , mc v̄(c) ∈ Rr×c and v̄(k)
is the average vector of vi .
(k) Thus, α is the eigenvector of eigen-problem Eqn. (6)
To calculate the c leading eigenvectors of HH T , it with the same eigenvalue λ.
is not necessary to work on matrix HH T which is of Theorem 1 shows that instead of solving the eigen-
size r × r. We can use a much more efficient algorithm. problem Eqn. (6), the KDA projective functions can be
Suppose the Singular Value Decomposition of H is obtained through two steps:

H = P Γ QT , 1. Solve the eigen-problem in Eqn. (10) to get y.

2. Find α which satisfies Kα α = y. The kernel matrix
it is easy to check that the column vectors of P are K is positive semi-definite. When K is non-singular
the eigenvectors of HH T and the column vectors of Q (positive definite), for any given y, we have a unique
are the eigenvectors of H T H [32]. Moreover, if P or α = K −1 y which satisfies the above linear equations
Q is given, we can recover the other via the formula system. When K is singular, the system may have
HQ = P Γ and P T H = Γ QT . Since c ≪ r, we can no solution or have infinite many solutions (the lin-
calculate the c eigenvectors of H T H and then recover ear equations system is underdetermined) [17]. A
the eigenvectors of HH T , which are β ’s. possible way is to approximate α by solving the fol-
We use the term flam [31], a compound operation lowing linear equations:
consisting of one addition and one multiplication, to α=y
(K + δI)α (11)
measure the operation counts. All the kernel methods
need to compute the kernel matrix K which requires where I is the identity matrix and δ ≥ 0 is the
O(m2 n) flam, where n is the number of features. The regularization parameter.
eigen-decomposition of K requires 92 m3 flam [32, 17]; The advantages of this two-step approach are as fol-
Calculating the c − 1 eigenvectors β ’s requires 92 c3 + lows:
3 2 2
α β
2 mc flam; Computing ’s from ’s requires m c flam.
Finally, we conclude the time complexity of KDA mea- 1. We will show later how the eigen-problem in Eqn.
sured by flam is (10) is trivial and we can directly get those eigen-
vectors y.
9 3 3 9 2. The eigen-decomposition of K is avoided. Since the
m + m2 c + O(m2 n) + mc2 + c3 .
2 2 2 matrix K + δI is positive definite, the Cholesky de-
composition can be used to efficiently solve the lin-
Considering m ≫ c, the above time complexity can be ear equations in Eqn. (11) [17], [31]. The computa-
simplified as tional complexity analysis will be provided in the
later section.
9 3
m + m2 c + O(m2 n). (9) The linear equations system in Eqn. (11) has a close
2
connection with regularized regression [36]. We denote
For a large scale problem, we have m ≫ n. Thus, the the projective function in the feature space as:
time complexity of KDA is determined by 29 m3 , which m
is the cost of eigen-decomposition of size m × m kernel X
f (x) = hνν , φ(x)i = αi K(x, xi )
matrix K. i=1
6

It can be easily verified that the solution α ∗ = (K + matrix K is positive definite and the δ = 0, Theorem 1
δI)−1 y given by equations in Eqn. (11) is the optimal shows that the c − 1 solutions α k = K −1 yk are exactly
solution of the following regularized regression problem the eigenvectors of the KDA eigen-problem in Eqn. (6)
[36]: with respect to the eigenvalue 1. In this case, SRKDA is
m equivalent to ordinary KDA. Thus, it is interesting and
X
2
min f (xi ) − yi + δkf k2K (12) important to see when the positive semi-definite kernel
f ∈F matrix K will be positive definite.
i=1
One of the most popular kernels is the Gaussian
where yi is the i-th element of y, F is the RKHS as-
RBF kernel, K(xi , xj ) = exp(−kxi − xj k2 /2σ 2 ). Our
sociated with Mercer kernel K and k kK is the corre-
discussion in this section will only focus on Gaussian
sponding norm.
kernel. Regarding the Gaussian kernel, we have the fol-
Now let us analyze the eigenvectors of W which is lowing lemma:
defined in Eqn. (7) and (8). The W is block-diagonal, Lemma 1 (Full Rank of Gaussian RBF Gram
thus, its eigenvalues and eigenvectors are the union of Matrices [21]) Suppose that x1 , · · · , xm are distinct
the eigenvalues and eigenvectors of its blocks (the latter points, and σ 6= 0. The matrix K given by
padded appropriately with zeros). It is straightforward
to show that W (k) has eigenvector e(k) ∈ Rmk asso- Kij = exp(−kxi − xj k2 /2σ 2 )
ciated with eigenvalue 1, where e(k) = [1, 1, · · · , 1]T . has full rank.
Also there is only one non-zero eigenvalue of W (k) be-
Proof See [21] and Theorem 2.18 in [27].
cause the rank of W (k) is 1. Thus, there are exactly c
eigenvectors of W with the same eigenvalue 1. These In other words, the kernel matrix K is positive definite
eigenvectors are (provided no two xi are the same).
Thus, we have the following theorem:
yk =[ 0, · · · , 0 , 1, · · · , 1, 0, · · · , 0 ]T k = 1, · · · , c (13) Theorem 2 If all the sample vectors are different and
| {z } | {z } | {z }
Pk−1 Pc
mi mk i=k+1 mi the Gaussian RBF kernel is used, all c − 1 projective
i=1

functions in SRKDA are eigenvectors of eigen-problem

Since 1 is a repeated eigenvalue of W , we can just pick
in Eqn. (6) with respect to eigenvalue 1 when δ = 0. In
any other c orthogonal vectors in the space spanned by
other words, the SRKDA and ordinary KDA are equiv-
{yk }, and define them to be our c eigenvectors. The
alent.
vector of all ones e is naturally in the spanned space.
This vector is useless since the corresponding projec- Proof This theorem can be easily proofed by combining
tive function will embed all the samples to the same Lemma 1 and Theorem 1.
point. Therefore, we pick e as our first eigenvector of It is easy to check that the values of the i-th and j-
W and use Gram-Schmidt process to orthogonalize the th entries of any vector y in the space spanned by {yk }
remaining eigenvectors. The vector e can then be re- in Eqn. (13) are the same as long as xi and xj belong
moved, which leaves us exactly c − 1 eigenvectors of W . to the same class. Thus the i-th and j-th rows of Ȳ are
We denote them as: the same, where Ȳ = [ȳ1 , · · · , ȳc−1 ]. Theorem (1) shows
c−1 T T that when the kernel matrix is positive definite, the c−1
{ȳk }k=1 , (ȳk e = 0, ȳi ȳj = 0, i 6= j) (14)
projective functions of KDA are exactly the solutions
The above two-step approach essentially combines of the c − 1 linear equations systems Kα αk = ȳk . Let
the spectral analysis of the matrix W and regression Θ = [αα1 , · · · , α c−1 ] be the KDA transformation matrix
techniques. Therefore, we name this new approach as which embeds the data points into the KDA subspace
Spectral Regression Kernel Discriminant Analysis (SRKDA). as:
In the following several subsections, we will provide ΘT [K(:, x1 ), · · · , K(:, xm )] = Ȳ T .
the theoretical and computational analysis on SRKDA.
Please see [6] for applying the similar technique on Lin- The columns of matrix Ȳ T are the embedding results
ear Discriminant Analysis to obtain an efficient algo- of data samples in the KDA subspace. Thus, the data
rithm. points with the same label are corresponding to the
same point in the KDA subspace when the kernel ma-
trix K is positive definite.
3.1 Theoretical Analysis These projective functions are optimal in the sense
of separating training samples with different labels. How-
SRKDA calculates the projective functions through the ever, they usually overfit the training set thus may not
linear equations system in Eqn. (11). When the kernel be able to perform well for the test samples.
 7

3.2 Computational Analysis

The computation of SRKDA involves two steps: re-

sponses (ȳk in Eqn. 14) generation and regularized re-
gression. The cost of the first step is mainly the cost
of Gram-Schmidt method, which requires (mc2 − 13 c3 )
flam [31].
To solve the c − 1 linear equations systems in Eqn.
(11), we can use the Cholesky decomposition, which Fig. 1 Sherman’s march (Cholesky decomposition)
uniquely factorizes the positive definite matrix K + δI
in the form K +δI = RT R, where R is upper triangular
with positive diagonal elements. The Cholesky decom- been computed with R and RT separated by a diag-
position requires 16 m3 flam [31]. With this Cholesky onal line1 . The white area represents untouched ele-
decomposition, the c − 1 linear equations can be solved ments of the original matrix. The thin vertical box rep-
within m2 c flam [31]. Besides solving the SRKDA opti- resents the column of R about to be computed. The
mization problem, we also need to compute the kernel algorithm is easy to derive. We show how to proceed
matrix K which requires O(m2 n) flam, where n is the from (m − 1) × (m − 1) submatrix to a m × m matrix.
number of features. Thus, the computational cost of We have
 
SRKDA is Km−1 k1m
Km =
1 3 1 kT1m kmm
m + m2 c + O(m2 n) + mc2 − c3 ,  T  
6 3 Rm−1 0 Rm−1 r1m
= ,
rT1m rmm 0 rmm
which can be simplified as
which leads to
1 3
m + m2 c + O(m2 n). T
Km−1 =Rm−1 Rm−1
6
T
k1m =Rm−1 r1m
Comparing to the computational cost of ordinary KDA
in Eqn. (9), SRKDA reduces the dominant part, which kmm =rT1m r1m + rmm
2

is 92 m3 of ordinary KDA, to 16 m3 ; achieves a 27-times When the Cholesky decomposition of the (m − 1) ×

speedup. (m − 1) submatrix Km−1 is known, it is easy to get the
Cholesky decomposition of the m × m Km . For detailed
derivation, please see [31].
4 Incremental KDA via Spectral Regression Now, let us consider the additional computational
cost of incremental SRKDA when ∆m new data sam-
Due to the difficulty of designing an incremental so- ples are injected to the system which already has m
lution for the eigen-decomposition on the kernel ma- samples. Comparing to the batch mode of SRKDA, we
trix in KDA, there has been little work on designing can get computational saving on two steps:
incremental KDA algorithms that can efficiently incor-
porate new data examples as they become available. 1. We only need to calculate the additional part of ker-
The SRKDA algorithm uses regression instead of eigen- nel matrix which requires O(nm∆m + n∆m2 ) flam;
decomposition to solve the optimization problem, which 2. The incremental Cholesky decomposition requires
1 3 1 3
provides us the chance to develop incremental version 6 (m + ∆m) − 6 m flam [31].
of SRKDA. Thus, the computation cost of incremental SRKDA mea-
The major cost in SRKDA computation is the step sured by flam is
of Cholesky decomposition which requires 61 m3 flam.
1 2 1 1
Fortunately, the Cholesky decomposition can be easily m ∆m + m∆m2 + ∆m3 + (m + ∆m)2 c
implemented in the incremental manner [31]. Actually, 2 2 6
1
Sherman’s march, one of the most popular Cholesky + O(nm∆m + n∆m ) + (m + ∆m)c2 − c3 .
2

decomposition algorithms, is implemented in the incre- 3

mental manner [31]. When ∆m ≪ m and c ≪ m, the above cost can be
The procedure of Sherman’s march is illustrated simplified as
graphically in Figure 1. The gray area represents the ∆m
part of the Cholesky decomposition that has already ( + c)m2 + O(nm∆m).
2
8

Table 2 Computational complexity of KDA and SRKDA Zou et al . [37] proposed an elegant sparse PCA al-
Algorithm operation counts (flam [31]) gorithm (SPCA) using their “Elastic Net” framework
KDA 9 3
m + cm2 + O(nm2 )
for L1 -penalized regression on regular principle com-
Batch mode 2
ponents, solved very efficiently using least angle regres-
1 3
SRKDA 6
m + cm2 + O(nm2 ) sion (LARS) [13]. Subsequently, d’Aspremont et al . [11]
Incremental KDA 9 3
2
m + cm2 + O(nm∆m) relaxed the hard cardinality constraint and solved for
mode a convex approximation using semi-definite program-
SRKDA ( ∆m
2
+ c)m2 + O(nm∆m)
ming. In [24, 25], Moghaddam et al . proposed a spectral
m: the number of data samples
bounds framework for sparse subspace learning. Partic-
n: the number of features
c: the number of classes ularly, they proposed both exact and greedy algorithms
∆m: the number of new data samples for sparse PCA and sparse LDA.
The projective function of a kernel subspace learn-
ing algorithm can be written as
m
X
We summarize our complexity analysis results in
Table 2. The main conclusions include: f (x) = α T K(:, x) = αi K(xi , x). (15)
i=1

– The ordinary KDA needs to perform eigen-decompositionIn the ordinary kernel subspace learning algorithms, αi
on the kernel matrix, which is very computation- are usually nonzero and the projective function is de-
ally expensive. Moreover, it is difficult to develop pendent on all the samples in the training set. When
incremental algorithm based on the ordinary KDA we aim at learning a sparse function (sparse α), many
formulation. In both batch and incremental modes, αi will equal to zero. Thus, the projective function will
ordinary KDA has the dominant part of the cost as only depend on part of the training samples. From this
9 3
2m . sense, the sparse kernel subspace learning algorithms
– SRKDA performs regression instead of eigen-decomposition.
share the similar idea of Support Vector Machines [36].
In the batch mode, it only has the dominant part Those samples with non-zero αi can also be called as
of the cost as 16 m3 , which is a 27-times speedup of support vectors. One advantage of this parsimony is
ordinary KDA. Moreover, it is easy to develop incre- that it requires less storage for the model and less com-
mental version of SRKDA which only has quadratic- putational time in the testing phase.
time complexity with respect to m. This computa- Following [25], the objective function of Sparse Ker-
tional advantage makes SRKDA much more practi- nel Discriminant Analysis (SparseKDA) can be defined
cal in real world applications. as the following cardinality-constrained optimization:
T
α
max αα TKW Kα
KKαα (16)
α) = k
subject to card(α
5 Sparse KDA via Spectral Regression
The feasible set is all sparse α ∈ Rm with k non-zero
Since SRKDA uses regression as a building block, vari- elements and card(α α) as their L0 -norm. Unfortunately,
ous kinds of regularization techniques can be easily in- this optimization problem is NP-hard and generally in-
corporated which makes SRKDA more flexible. In this tractable .
section, we will discuss the usage of L1 -norm regularizer In [24, 25], Moghaddam et al . proposed a spectral
to produce a sparse KDA solution. bounds framework for sparse subspace learning. Partic-
Recently, there are considerable interests on devel- ularly, they proposed both exact and greedy algorithms
oping sparse subspace learning algorithms, i.e., the pro- for sparse PCA and sparse LDA. Their spectral bounds
jective vectors are sparse. While the traditional linear framework is based on the following optimal condition
subspace learning algorithms (e.g., PCA, LDA) learn of the sparse solution.
a set of combined features which are linear combina- For simplicity, we define A = KW K and B = KK.
tions of all the original features, sparse linear subspace A sparse vector α ∈ Rm with cardinality k yielding the
learning algorithms can learn the combined features maximum objective value in Eqn. (16) would necessar-
which are linear combinations of part of the original ily imply that
features (important ones). Such parsimony not only pro- α T Aα
α β T Akβ
duces a set of projective functions that are easy to in- λmax = T
= T
α Bα α β Bk β
terpret but also leads to better performance [37, 25].
where β ∈ Rk contains the k non-zero elements in α and
1 Actually, we only need to store R. the k × k principle sub-matrices of A and B obtained
 9

by deleting the rows and columns corresponding to the Table 3 Statistics of the three data sets
zero indices of α . The k-dimensional quadratic form in train test # of
dataset dim (n)
β is equivalent to a standard unconstrained generalized size (m) size classes (c)
Rayleigh quotient, which can be solved by a generalized Isolet 617 6238 1559 26
USPS 256 7291 2007 10
eigen-problem.
PIE 1024 8000 3554 68
The above observation gives the exact algorithm for
sparse subspace learning: a discrete search for the k in-
dices which maximize λmax of the subproblem (Ak , Bk ).
However, such observation does not suggest an efficient all the possible cardinality on α ) of the regression prob-
algorithm because an exhaustive search is still NP-hard. lem in Eqn. (17) can be computed in O(m3 ). Thus,
To solve this problem, Moghaddam et al . proposed an SRKDA with a L1 -norm regularizer provides us an ef-
efficient greedy algorithm which combines backward elim- ficient algorithm to compute the sparse KDA solution.
ination and forward selection [24, 25]. However, there
are two major drawbacks of their approach:
6 Experimental Results
1. Even their algorithm is a greedy one, the cost of
backward elimination is with complexity O(m4 )[25].
In this section, we investigate the performance of our
2. In reality, more than one projective functions are
proposed SRKDA algorithm in batch mode, incremen-
usually necessary for subspace learning. However,
tal mode and sparse mode. All of our experiments have
the optimal condition of the sparse solution only
been performed on a P4 3.20GHz Windows XP machine
gives the guide to find ONE sparse “eigenvector”,
with 2GB memory. For the purpose of reproducibility,
which is the first projective function. It is unclear
we provide all the algorithms used in these experiments
how to find the following projective functions. Al-
at:
though [24] suggests to use recursive deflation, the
http://www.zjucadcg.cn/dengcai/Data/data.html
sparseness of the following projective functions is
not guaranteed.

Our SRKDA algorithm uses the regression instead 6.1 Datasets

of eigen-decomposition to solve the optimization prob-
lem. Thus, it provides us the chance to develop more Three datasets are used in our experimental study, in-
efficient sparse KDA algorithm. cluding spoken letter, handwritten digit image, and face
Recall the second step of SRKDA, which is solving image data sets. The important statistics of three datasets
the linear equations system Kα α = y. Essentially, we try are summarized below (see also Table 3):
to solve a regression problem:
– The Isolet spoken letter recognition database2 was
m 
X 2 first used in [14]. It contains 150 subjects who spoke
min K(:, xi )T α − yi the name of each letter of the alphabet twice. The
α
i=1
speakers are grouped into sets of 30 speakers each,
where K(:, xi ) is the i-th column of K and yi is the i-th and are referred to as isolet1 through isolet5. In the
element of y. We can add different regularizers to get past usage [14][12], isolet1&2&3&4 were used as the
different solutions with desired properties. The SRKDA training set and isolet5 was used as the test set.
algorithm we described in the previous section essen- For the purposes of our experiment, we also choose
tially adds a L2 -norm regularizer. We can also use a isolet5 as the test set and perform several runs with
L1 -norm regularizer: isolet1, isolet1&2, isolet1&2&3, and isolet1&2&3&4
m 
! as the training set respectively.
X 2 Xm
min K(:, xi )T α − yi + δ |αi | (17) – The USPS handwritten digit database is described
α
i=1 i=1 in [19]. A popular subset 3 contains 9298 16 × 16
handwritten digit images in total, which is then split
which is usually referred as lasso regression [18]. Due to
into 7291 training images and 2007 test images. In
the nature of the L1 penalty, some coefficients αi will
our experiment, we train all the algorithms on the
be shrunk to exact zero if δ is large enough. Therefore,
first 1500 (3000, 4500, 6000, and 7291) images in the
the lasso produces a sparse projective function, which
training set and test on the 2007 test images.
is exactly what we want.
By using the Least Angel Regression (LARS) algo- 2 http://www.ics.uci.edu/∼mlearn/MLSummary.html
rithm [13], the entire solution path (the solutions with 3 http://www.csie.ntu.edu.tw/∼cjlin/libsvmtools/datasets/multiclass.html#usps
10

– The CMU PIE face database4 contains 68 subjects The main observations from the performance com-
with 41,368 face images as a whole. The face images parisons include:
were captured under varying pose, illumination and – The Kernel Discriminant Analysis model is very ef-
expression. In our experiment, the five near frontal fective in classification. SRKDA has the best per-
poses (C05, C07, C09, C27, C29) under different il- formance for almost all the cases in all the three
luminations and expressions are used which leaves data sets (even better than SVM). For Isolet data
us 11,554 face images. All the images are manually set, previous study [12] reported the minimum error
aligned and cropped. The cropped images are 32×32 rate training on Isolet1+2+3+4 by OPT6 with 30
pixels, with 256 gray levels per pixel5 . Among the bit ECOC is 3.27%. KDA (SRKDA) achieved better
11,554 images, 8,000 images are used as the train- performance in our experiment for this train/test
ing set and the remaining 3,554 images are used for split. For USPS data set, previous studies [27] re-
testing. We also run several cases by training all the ported error rate 3.7% for KDA and 4.0% for SVM,
algorithms on the first 2000, 3000, · · · , 8000 images slightly better than the results in our experiment.
in the training set. For all the cases, KDA (SRKDA) achieved signif-
icantly better performance than LDA, which sug-
6.2 Compared algorithms gests the effectiveness of kernel approaches.
– Since the eigen-decomposition of the kernel matrix
Four algorithms which are compared in our experiments is involved, the ordinary KDA is computationally
are listed below: expensive in training. SRKDA uses regression in-
stead of eigen-decomposition to solve the optimiza-
1. Linear Discriminant Analysis (LDA) [15], which pro- tion problem, and thus achieves significant speedup
vides us a baseline performance of linear algorithms. comparing to ordinary KDA. The empirical results
We can examine the usefulness of kernel approaches are consistent with the theoretical estimation of the
by comparing the performance of KDA and LDA. efficiency. The time of training SRKDA is compa-
2. Kernel Discriminant Analysis (KDA) as discussed rable with that of training SVM. SRKDA is faster
in Section 2. We test the regularized version and than SVM on Isolet and PIE data sets, while slower
choose the regularization parameter δ by five fold than SVM on USPS data set. This is because the
cross-validation on the training set. time of training SVM is dependant with the number
3. Spectral Regression Kernel Discriminant Analysis of support vectors [2]. For some data sets with lots
(SRKDA), our approach proposed in this paper. of noise (e.g., USPS), the number of support vectors
The regularization parameter δ is also chosen by is far less than the number of samples. In this case,
five fold cross-validation on the training set. SVM can be trained very fast.
4. Support Vector Machine (SVM) [36], which is be-
lieved as one of the state-of-the-art classification al-
gorithms. Specifically, we use the LibSVM system 6.4 Experiments on Incremental KDA
[10] which implements the multi-class classification
with one versus one strategy. SVM is used to get the In this experiment, we study the computational cost
sense that how good the performance of KDA is. of SRKDA performing in the incremental manner. The
USPS and PIE data sets are used. We start from the
We use the Gaussian RBF kernel for all the kernel-
training set with the size of 1000 (the first 1000 sam-
based methods. We tune the kernel width parameter σ
ples in the whole training set) and increase the training
and large margin parameter C in SVM to achieve best
size by 200 for each step. SRKDA is then performed in
testing performance for SVM. Then, the same kernel
the incremental manner. It is important to note that
width parameter σ is used in all the other kernel-based
SRKDA in the incremental manner gives the exactly
algorithms.
same projective functions as the SRKDA in the batch
mode. Thus, we only care about the computational
6.3 Results costs in this experiment.
Figure 2 and 3 shows log-log plots of how CPU-time
The classification error rate as well as the training time of KDA (SRKDA, incremental SRKDA) increases with
(second) for each method on the three data sets are the size of the training set on USPS and PIE data set
reported on the Table (4 ∼ 6) respectively. respectively. Lines in a log-log plot correspond to poly-
4
nomial growth O(md ), where d corresponds to the slope
http://www.ri.cmu.edu/projects/project 418.html
5 http://www.zjucadcg.cn/dengcai/Data/FaceData.html 6 Conjugate-gradient implementation of back-propagation
 11

Table 4 Performance comparisons on Isolet dataset

Error (%) Time (s)
Speedup
Training Set LDA KDA SRKDA SVM LDA KDA SRKDA SVM
Isolet1 15.27 11.74 12.89 12.51 1.93 18.86 1.21 4.75 15.6
Isolet1+2 6.61 3.79 3.85 4.11 2.14 134.6 5.51 13.79 24.4
Isolet1+2+3 5.90 2.99 3.08 3.34 2.37 451.6 14.09 23.84 32.1
Isolet1+2+3+4 5.71 2.82 2.89 3.27 2.56 991.2 27.86 34.82 35.6
∗ Column labeled “Speedup” shows how many times faster the SRKDA is

(comparing to ordinary KDA).

Table 5 Performance comparisons on USPS dataset

Error (%) Time (s)
Speedup
Training Set LDA KDA SRKDA SVM LDA KDA SRKDA SVM
1500 10.61 6.58 5.88 6.85 0.21 14.97 0.92 0.78 16.3
3000 9.77 5.53 5.38 5.58 0.27 111.9 4.35 2.20 25.7
4500 9.52 5.53 4.88 5.13 0.34 354.3 11.29 4.06 31.4
6000 9.92 5.03 4.43 5.08 0.40 825.3 22.74 6.22 36.3
7291 10.26 4.83 4.04 4.83 0.47 1553.6 37.59 8.18 41.3

Table 6 Performance comparisons on PIE dataset

Error (%) Time (s)
Speedup
Training Set LDA KDA SRKDA SVM LDA KDA SRKDA SVM
2000 5.29 5.18 4.81 6.30 8.77 36.51 2.47 24.13 14.8
3000 4.61 4.25 3.94 4.70 9.06 116.9 5.39 43.99 21.7
4000 4.14 5.53 3.24 3.74 9.42 256.6 10.35 68.43 24.8
5000 3.85 3.23 2.90 3.29 9.73 502.3 17.40 96.26 28.9
6000 3.57 2.91 2.53 2.84 10.06 830.7 27.21 125.6 30.5
7000 3.40 2.65 2.19 2.64 10.39 1340.9 38.65 155.6 34.7
8000 3.35 2.41 2.17 2.34 10.79 1908.1 53.75 186.7 35.5

4
of the line. The ordinary KDA scales roughly O(m2.9 ), 10
KDA
which is slightly better than the theoretical estimation. SRKDA
SRKDA−Incre
SRKDA in the batch mode has better scaling, which 3
10
O(m2.9)
is also better than theoretical estimation with roughly
Computational time (s)

O(m2.6)
O(m2.6 ) over much of the range. This explains why 2
10
1.8
O(m )
SRKDA can be more than 27 times faster than ordi-
nary KDA in the previous experiments. The SRKDA 1
10
in the incremental mode has the best scaling, which is
(to some surprise) better than quadratic with roughly
0
O(m1.8 ) over much of the range. 10

−1
10
3 4
10 10
6.5 Experiments on Sparse KDA Training size

Fig. 2 Computational cost of KDA, batch SRKDA and incre-

In this experiment, we study the performance of SRKDA mental SRKDA on the USPS data set.
performing in the sparse mode, i.e., the SRKDA with
L1 -norm regularizer to produce the sparse KDA solu-
tion. To the best of our knowledge, there is no other Table (7), (8) and (9) show the classification error
published method to generate a sparse KDA solution. rate of SRKDA in sparse mode on the three data sets re-
Moghaddam’s sparse LDA approach [25] can be mod- spectively. By using the Least Angel Regression (LARS)
ified to generate the sparse KDA solution. However, algorithm [13], the entire solution path (the solutions
as we pointed out in the last section, their approach with all the possible cardinality on the projective func-
can only generate ONE sparse projective function and tion α ) can be computed. After this, we use cross vali-
is only suitable for binary class problem, while all the dation to select the optimal cardinality of the projective
three data sets studied in this paper are multi-class data function in the experiment. We also show the sparsity
sets. of the projective function of SRKDA(sparse) in the ta-
12

4
10 formance of SRKDA(sparse) is better than that of the
KDA
SRKDA ordinary KDA and SRKDA.
3 SRKDA−Incre
10 2.9
O(m )
Computational time (s)

O(m2.6)
2
10 O(m1.8) 7 Conclusions

1
In this paper, we propose a novel algorithm for kernel
10
discriminant analysis, called Spectral Regression Ker-
nel Discriminant Analysis (SRKDA). Our algorithm is
0
10 developed from a graph embedding viewpoint of KDA
problem. It combines the spectral graph analysis and
−1
10 regression to provide an efficient approach for kernel
3 4
10 10 discriminant analysis. Specifically, SRKDA only needs
Training size
to solve a set of regularized regression problems and
Fig. 3 Computational cost of KDA, batch SRKDA and incre- there is no eigenvector computation involved, which is
mental SRKDA on the PIE data set.
a huge save of computational cost. The theoretical anal-
ysis shows that SRKDA can achieve 27-times speedup
Table 7 Classification error on Isolet dataset over the ordinary KDA. Moreover, the new formulation
Error (%) Sparsity makes it very easy to develop incremental version of the
Training Set KDA SRKDA SRKDA(Sparse) algorithm which can fully utilize the computational re-
Isolet1 11.74 12.89 11.74 60% sults of the existing training samples. With incremental
Isolet1+2 3.79 3.85 3.59 60% implementation, the computational cost of SRKDA re-
Isolet1+2+3 2.99 3.08 2.82 60%
Isolet1+2+3+4 2.82 2.89 2.82 60%
duces to quadratic-time complexity. Since SRKDA uses
regression as a building block, various kinds of regular-
Table 8 Classification error on USPS dataset ization techniques can be easily incorporated (e.g., L1 -
norm regularizer to produce sparse projections). Our
Error (%) Sparsity
Training Set KDA SRKDA SRKDA(Sparse) approach provides a huge possibility to develop new
1500 6.58 5.88 5.83 60% variations of kernel discriminant analysis. Extensive ex-
3000 5.53 5.38 5.13 60% perimental results show that our method consistently
4500 5.53 4.88 4.73 60%
outperforms the other state-of-the-art KDA extensions
6000 5.03 4.43 4.04 60%
7291 4.83 4.04 3.94 60% considering both effectiveness and efficiency.

Table 9 Classification error on PIE dataset Acknowledgements This work was supported in part by Na-
tional Natural Science Foundation of China under Grants 60905001
Error (%) Sparsity
and 90920303, National Key Basic Research Foundation of China
Training Set KDA SRKDA SRKDA(Sparse)
under Grant 2009CB320801. Any opinions, findings, and conclu-
2000 5.18 4.81 4.73 60%
sions expressed here are those of the authors and do not neces-
3000 4.25 3.94 3.71 60%
sarily reflect the views of the funding agencies.
4000 5.53 3.24 3.12 60%
5000 3.23 2.90 2.81 60%
6000 2.91 2.53 2.44 60%
7000 2.65 2.19 2.17 60% References
8000 2.41 2.17 2.14 60%
1. G. Baudat and F. Anouar. Generalized discriminant analysis
using a kernel approach. Neural Computation, (12):2385–
2404, 2000.
bles. The sparsity is defined as the percentage of zero 2. C. J. C. Burges. A tutorial on support vector machines for
entries in a projective vector. For ordinary KDA and pattern recognition. Data Mining and Knowledge Discovery,
2(2):121–167, 1998.
SRKDA, the projective functions (vectors) are dense 3. D. Cai, X. He, and J. Han. Efficient kernel discriminant
and the sparsity is zero. analysis via spectral regression. In Proc. Int. Conf. on Data
Mining (ICDM’07), 2007.
As can be seen, the SRKDA(sparse) generates much 4. D. Cai, X. He, and J. Han. Spectral regression: A unified
more parsimonious model. The sparsity of the projec- approach for sparse subspace learning. In Proc. Int. Conf.
tive function in SRKDA(sparse)is 60%, which means on Data Mining (ICDM’07), 2007.
5. D. Cai, X. He, and J. Han. Spectral regression: A unified sub-
the number of the “support vectors” is less than half of
space learning framework for content-based image retrieval.
the total training samples. Moreover, such parsimony In Proceedings of the 15th ACM International Conference
leads to better performance. In all the cases, the per- on Multimedia, Augsburg, Germany, 2007.
 13

6. D. Cai, X. He, and J. Han. SRDA: An efficient algorithm the 2005 ACM SIGMOD international conference on Man-
for large scale discriminant analysis. IEEE Transactions on agement of data, pages 730–741, 2005.
Knowledge and Data Engineering, 20(1):1–12, January 2008. 29. H. T. Shen, X. Zhou, and B. Cui. Indexing text and visual
7. D. Cai, X. He, W. V. Zhang, and J. Han. Regularized locality features for www images. In 7th Asia Pacific Web Confer-
preserving indexing via spectral regression. In Proceedings of ence (APWeb2005), 2005.
the 16th ACM conference on Conference on information and 30. H. T. Shen, X. Zhou, and A. Zhou. An adaptive and dynamic
knowledge management (CIKM’07), pages 741–750, 2007. dimensionality reduction method for high-dimensional index-
8. K. Chakrabarti and S. Mehrotra. Local dimensionality reduc- ing. The VLDB Journal, 16(2):219–234, 2007.
tion: A new approach to indexing high dimensional spaces. 31. G. W. Stewart. Matrix Algorithms Volume I: Basic Decom-
In Proc. 2000 Int. Conf. Very Large Data Bases (VLDB’00), positions. SIAM, 1998.
2000. 32. G. W. Stewart. Matrix Algorithms Volume II: Eigensystems.
9. S. Chakrabarti, S. Roy, and M. V. Soundalgekar. Fast and SIAM, 2001.
accurate text classification via multiple linear discriminant 33. D. Tao, X. Li, X. Wu, and S. J. Maybank. General tensor
projections. The VLDB Journal, 12(2):170–185, 2003. discriminant analysis and gabor features for gait recognition.
10. C.-C. Chang and C.-J. Lin. LIBSVM: a library for IEEE Transactions on Pattern Analysis and Machine Intel-
support vector machines, 2001. Software available at ligence, 29(10):1700–1715, 2007.
http://www.csie.ntu.edu.tw/∼cjlin/libsvm. 34. D. Tao, X. Li, X. Wu, and S. J. Maybank. Geometric mean
11. A. d’Aspremont, L. E. Chaoui, M. I. Jordan, and G. R. G. for subspace selection. IEEE Transactions on Pattern Anal-
Lanckriet. A direct formulation for sparse PCA using ysis and Machine Intelligence, 31(2):260–274, 2009.
semidefinite programming. In Advances in Neural Informa- 35. D. Tao, X. Tang, X. Li, and Y. Rui. Kernel direct biased
tion Processing Systems 17, 2004. discriminant analysis: A new content-based image retrieval
12. T. G. Dietterich and G. Bakiri. Solving multiclass learning relevance feedback algorithm. IEEE Transactions on Multi-
problems via error-correcting output codes. Jounal of Arti- media, 8(4):716–727, 2006.
ficial Intelligence Research, (2):263–286, 1995. 36. V. N. Vapnik. Statistical learning theory. John Wiley &
13. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least Sons, 1998.
angle regression. Annals of Statistics, 32(2):407–499, 2004. 37. H. Zou, T. Hastie, and R. Tibshirani. Sparse principle com-
14. M. A. Fanty and R. Cole. Spoken letter recognition. In ponent analysis. Technical report, Statistics Department,
Advances in Neural Information Processing Systems 3, 1990. Stanford University, 2004.
15. K. Fukunaga. Introduction to Statistical Pattern Recogni-
tion. Academic Press, 2nd edition, 1990.
16. V. Gaede and O. Günther. Multidimensional access methods.
ACM Comput. Surv., 30(2):170–231, 1998.
17. G. H. Golub and C. F. V. Loan. Matrix computations. Johns
Hopkins University Press, 3rd edition, 1996.
18. T. Hastie, R. Tibshirani, and J. Friedman. The Elements
of Statistical Learning: Data Mining, Inference, and Predic-
tion. New York: Springer-Verlag, 2001.
19. J. J. Hull. A database for handwritten text recognition re-
search. IEEE Trans. Pattern Anal. Mach. Intell., 16(5),
1994.
20. H. Jin, B. C. Ooi, H. T. Shen, C. Yu, and A. Zhou. An
adaptive and efficient dimensionality reduction algorithm for
high-dimensional indexing. In Proc. 2003 Int. Conf. on Data
Engineering (ICDE’03), 2003.
21. C. A. Micchelli. Algebraic aspects of interpolation. In Pro-
ceedings of Symposia in Applied Mathematics, volume 36,
pages 81–102, 1986.
22. S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K.-R.
Müller. Fisher discriminant analysis with kernels. In Proc.
of IEEE Neural Networks for Signal Processing Workshop
(NNSP), 1999.
23. S. Mika, A. Smola, and B. Schölkopf. An improved train-
ing algorithm for kernel fisher discriminants. In Proceedings
AISTATS 2001. Morgan Kaufmann, 2001.
24. B. Moghaddam, Y. Weiss, and S. Avidan. Spectral bounds
for sparse PCA: Exact and greedy algorithms. In Advances
in Neural Information Processing Systems 18, 2005.
25. B. Moghaddam, Y. Weiss, and S. Avidan. Generalized spec-
tral bounds for sparse LDA. In ICML ’06: Proceedings of the
23rd international conference on Machine learning, pages
641–648, 2006.
26. C. H. Park and H. Park. Nonlinear discriminant analysis
using kernel functions and the generalized singular value de-
composition. SIAM J. Matrix Anal. Appl., 27(1):87–102,
2005.
27. B. Schölkopf and A. J. Smola. Learning with Kernels. MIT
Press, 2002.
28. H. T. Shen, B. C. Ooi, and X. Zhou. Towards effective index-
ing for very large video sequence database. In Proceedings of