2014 International Joint Conference on Neural Networks (IJCNN)

July 6-11, 2014, Beijing, China

Kernel Ridge Regression Classification

Jinrong He, Lixin Ding, Lei Jiang and Ling Ma

Abstract—We present a nearest nonlinear subspace classifier subspace classification methods [9][10] classify a new test
that extends ridge regression classification method to kernel sample into the class whose subspace is the closest. CLASS
version which is called Kernel Ridge Regression Classification featuring information compression (CLAFIC) is one of the
(KRRC). Kernel method is usually considered effective in earliest and well-known subspace methods [11] and its
discovering the nonlinear structure of the data manifold. The
extension into the nonlinear subspace is the Kernel CLAFIC
basic idea of KRRC is to implicitly map the observed data into
potentially much higher dimensional feature space by using (KCLAFIC). This method employs the principal component
kernel trick and perform ridge regression classification in analysis to compute the basis vectors spanning subspace of
feature space. In this new feature space, samples from a each class. Linear Regression Classification (LRC) [12]
single-object class may lie on a linear subspace, such that a new method is another popular subspace method. In LRC,
test sample can be represented as a linear combination of classification is taken as a class specific linear regression
class-specific galleries, then the minimum distance between the
problem and the regression coefficients are estimated by
new test sample and class specific subspace is used for
classification. Our experimental studies on synthetic data sets using the least square estimation method, and then the
and some UCI benchmark datasets confirm the effectiveness of classification is made by the minimum distance between the
the proposed method. original sample and the projected sample. However, the least
square estimation is very sensitive to outliers [13]. Therefore,
the performance of LRC decreases sharply as the sample
I. INTRODUCTION contaminated by outliers. Due to L2,1-norm based loss

M ANY applications of machine learning, ranging from

text categorization to computer vision, require the
classification of large volumes of complex data sets. Among
function can reduce the effect of outliers, Ren Chuan-Xian
[14] proposed rotational invariant norm based regression
classification method. In 2012, Naseem et al. proposed a
the algorithms, the K nearest neighbor (KNN) method is one robust linear regression classification algorithm (RLRC) [15]
of the most successful and robust methods for many to estimate regression parameters by using the robust Huber
classification problems at the same time being simple and estimation. Compared with least square estimation, the
intuitive [1]. In this naive approach, a test sample is assigned Huber’s M-estimator weighs the large residuals more lightly.
to the class which contains the nearest sample. Despite its As a result, the outliers have less significant affection on the
advantages, the KNN algorithm suffers from poor estimated coefficients. Moreover, to overcome the problem of
generalization ability and becomes less effective when the multicollinearity in LRC, Huang and Yang [16] proposed an
samples having different class labels are comparable in the improved principal component regression classification
neighborhood of a test sample [2]. To overcome the (IPCRC) method which removes the mean of each sample
drawbacks of KNN, various methods have been proposed in before performing principal component analysis and drops
the literature [3][4][5][6]. the first principal components. The projected coefficients are
Subspace learning [7][8] is a traditional method for pattern then executed by the linear regression classification algorithm.
classification, which always assumes that data is sampled However, when the axes of linear regression of class-specific
from a linear subspace. Other than KNN method, nearest samples have an intersection, LRC could not well classify the
samples that distribute around the intersection. To improve
the performance of LRC in this situation, Yuwu Lu et al.
Jinrong He is with the School of Computer, State Key Laboratory of proposed a kernel linear regression classification (KLRC)
Software Engineering, Wuhan University, Wuhan, 430072 China
(corresponding author to provide phone: 086-15392822848; e-mail: algorithm [17], by integrating the kernel trick and LRC.
hejinrong@whu.edu.cn). Although many regression-based approaches have been
Lixin Ding is with State Key Laboratory of Software Engineering, Wuhan proposed to achieve successful classification tasks, LRC
University, Wuhan, 430072 China (e-mail: lxding@whu.edu.cn).
Lei Jiang is with Key Laboratory of Knowledge Processing and method fails when the number of sample in the class specific
Networked Manufacture, Hunan University of Science and Technology, training set is smaller than their dimension. The ridge
Xiangtan, 411201 China (e-mail: jlefe@126.com). regression method [18] is a regularized least square method
Ling Ma is with the Second Artillery Equipment Academy, Beijing 10085
China (e-mail: whjcc@163.com). for classification and regression. It is suitable only for
This work was supported in part by the Fundamental Research Funds for datasets with few training examples. Kernel methods [19] are
the Central Universities (No. 2012211020209), Special Project on the effective framework to enhance the modeling capability by
Integration of Industry, Education and Research of Ministry of Education and
Guangdong Province (2011B090400477), Special Project on the Integration nonlinearly mapping the data from the original space to a high
of Industry, Education and Research of Zhuhai City (2011A050101005, dimensional feature space which is called reproducing kernel
2012D0501990016), Zhuhai Key Laboratory Program for Science and Hilbert space (RKHS).
Technique (2012D0501990026).

978-1-4799-1484-5/14/$31.00 ©2014 IEEE 2263

Authorized licensed use limited to: INDIAN INST OF INFO TECH AND MANAGEMENT. Downloaded on December 05,2023 at 12:38:41 UTC from IEEE Xplore. Restrictions apply.
 In order to detect nonlinear structure of samples and If we take the derivative of Equation (2) with respect to α i
improve the robustness of LRC algorithm, we propose a and set it to zero, we get
Kernel Ridge Regression Classification (KRRC) method to X iT X iα i − X iT x + λα i = 0 (3)
boost the effectiveness of the LRC. KRRC is a nonlinear Then the estimate of the regression parameter vectors can
extension of ridge regression classification method based on be computed by
kernel trick, which implicitly maps the data into a 
α i = ( X iT X i + λ I ) X iT x
−1

high-dimensional kernel space by using a nonlinear mapping (4)

determined by a kernel function. KRRC method falls in the Thus the projection of x onto the subspace of the ith class
category of nearest subspace classification that shares similar can be computed as
 
x i = X iα i = X i ( X iT X i + λ I ) X iT xi  H i x
−1
idea with LRC. (5)
In the remainder of the paper, we will first describe the
Where H i is the class specific projection matrix which is
ridge regression classification method, and then the proposed
KRRC method is presented in Section II. It is followed by defined as follows:
H i = X i ( X iT X i + λ I ) X iT
−1
extensive experiments using synthetic data sets and UCI data (6)
sets in Section III. The paper concludes in Section IV. Note that the projection matrix is a symmetric matrix and
also idempotent, i.e., H iT = H i , H i2 = H i ⋅ H i = H i .
II. KERNEL RIDGE REGRESSION CLASSIFICATION After projecting new test sample onto every class-specific
subspace, the minimum distance between the new test sample
A. Ridge Regression Classification (RRC) and class specific subspace is used for classification. If the
original sample belongs to the subspace of class i, the
Ridge regression [20] is a classical data modeling method 
projected sample x i onto the class specific subspace Xi will
to solve multicollinearity problem of covariates in samples.
Here, multicollinearity refers to a situation in which more be the closet sample to the original sample.
 2
than one predictor variables in a multiple regression model i* = arg min x i − x
2
(7)
i
are highly correlated. If multicollinearity is perfect, the
regression coefficients are indeterminate and their standard Figure 1 shows geometric interpretation of LRC and RRC
errors are infinite. If it is less than perfect, the regression method. Ridge regression classification is a regularized least
coefficients although determinate but possess large standard square method to model the linear dependency between class
errors, which means that the coefficients cannot be estimated specific samples and the new test sample which can deal with
with great accuracy [21]. multicollinearity problem.
Using a fundamental concept that samples from a specific
class lie on a linear subspace, a new test sample from any x1i x1i
class can be represented as a linear combination of

class-specific training samples. This assumption can be xi 
xi
formulated as a linear model in terms of ridge regression. x2i
Assume that we have C classes and each class has ni x3i x2i
x3i
samples in the d-dimensional space. Let Xi be the training set
(a) LRC (b) RRC
of the ith class whose data matrix is Fig.1. Geometric Interpretation of LRC and RRC methods
X i = ⎡⎣ x1i , x2i ,", xni i ⎤⎦ ∈ R d ×ni
According to subspace assumption, the new test sample x B. Kernel Ridge Regression Classification
belongs to the ith class can be represented by the linear
The main idea of Kernel Ridge Regression Classification
combinations of these samples with an error ε according to (KRRC) is to map the original samples into a higher
LRC method. Hence dimensional Hilbert space F and apply RRC method on this
x = X iα i + ε (1) Hilbert space F. Kernel trick my increase the linearity of
Where α i is ni × 1 dimensional regression coefficients samples, i.e., a nonlinear curve can be taken as lying on a
vector. plane. The nonlinear mapping function can be denoted as
Similar to LRC, we formulated ridge regression φ : X → F . For any new test sample x, the goal of kernel ridge
classification method as follows. 
regression is to find α i to minimize the residual error as:
For any new test sample x, the goal of the ridge regression  2 2

is to find α i to minimize the residual error as: α ii = arg min φ ( x) − φ ( X i )α i 2 + λ α i 2
(8)
αi
 2 2
Similar to Equation (4), the estimate of the regression
α ii = arg min x − X iα i 2 + λ α i 2
(2)
αi
parameter vectors can be computed by
Here, λ is regularization parameter. Ridge regression can 
α i = (φ T ( X i )φ ( X i ) + λ I ) φ T ( X i )φ ( x)
−1
(9)
reduce the variance by penalizing the norm of the linear
transform and balance the bias and variance by adjusting the Then we can predict the response vector φq
i
( x) for the ith
regularization parameter. class as

2264
Authorized licensed use limited to: INDIAN INST OF INFO TECH AND MANAGEMENT. Downloaded on December 05,2023 at 12:38:41 UTC from IEEE Xplore. Restrictions apply.
 
φq
T
( x) = φ ( X i )α i φq
i i
( x) φ ( x)
= φ ( X i ) (φ T ( X i )φ ( X i ) + λ I ) φ T ( X i )φ ( x)
−1
= φ T ( x)φ ( X i ) (φ T ( X i )φ ( X i ) + λ I ) φ T ( X i )φ ( x)
−1
(10)
(17)
 H iφφ ( x) = K ( x, X i ) ( K ( X i , X i ) + λ I ) K ( X i , x )
−1

Where φq i
( x ) is the projection of φ ( x) onto the subspace of = K ( x, X i ) A
the ith class by the class specific projection matrix which is Here
defined as follows: A = ( K ( X i , X i ) + λ I ) K ( X i , x)
−1
(18)
H i = φ ( X i ) (φ ( X i )φ ( X i ) + λ I ) φ ( X i )
φ T −1 T
(11) Since K ( X i , X i ) + λ I is positive definite, and its Cholsky
If the original sample belongs to the subspace of class i, the decomposition can be written as
predicted sample φq i
( x ) in kernel space F will be the closet K ( X i , X i ) + λ I = LT L (19)
sample to the original sample. Then the matrix A in Equation (18) can be efficiently
2
i * = arg min φq
i
( x) − φ ( x) computed by solving the following linear equation
i 2 (12) LT LA = K ( X i , x) (20)
T T
= φq
i
( x) φq
i
( x) − 2 * φq
i
( x) φ ( x) + φ T ( x)φ ( x) Note that the third term in Equation (12) has no effect on
According to Mercer’s theorem [19], the form of nonlinear classification results, since it has nothing to do with the class
function φ ( x) is not necessarily known explicitly and could information. Therefore, after neglecting the third term, we
be determined by a kernel function k : X × X → R which has have
T T
the following property: φq
i
( x) φq
i
( x) − 2 * φq
i
( x) φ ( x)
k ( xi , x j ) = φ T ( xi )φ ( x j ) (13) = AT K ( X i , X i ) A − 2 * K ( x, X i ) A
(21)
There are numerous types of kernel functions [19]. In our = AT ( K ( X i , X i ) − 2 * ( K ( X i , X i ) + λ I ) ) A
experiments, we adopt most popular Gaussian kernel that is = − AT ( K ( X i , X i ) + 2λ I ) A
given by
Equivalently, the classification process (12) can be
⎛ x −x 2
⎞ reformulated as
k ( xi , x j ) = exp ⎜ − 2 ⎟
i j
(14)
⎜ t ⎟ i* = arg max { AT ( K ( X i , X i ) + 2λ I ) A} (22)
⎝ ⎠ i

The parameter t is empirically set as the average Euclidean The KRRC algorithm is given in Algorithm 1.
distance of all training samples.
Obviously, the classification process (12) can be expressed Algorithm 1 Kernel Ridge Regression Classification
in terms of inner products between mapped training samples Input: training data matrix X_train, Label vector for training
in F. Let us define kernel matrix K whose elements is data L_train and testing data matrix X_test.
⎛ k ( x1i , x1j ) k ( x1i , x2j ) " k ( x1i , xnj j ) ⎞ Procedure:
⎜ ⎟ For each testing data sample x, predict its label as follows:
⎜ k ( x2i , x1j ) k ( x2i , x2j ) " k ( x2i , xnj j ) ⎟
K(Xi, X j ) = ⎜ ⎟ (15) Step 1. Compute the kernel matrix K(Xi,Xi) and K(Xi,x) with
⎜ # # " # ⎟ Gaussian kernel (14).
⎜ k ( xi , x j ) k ( xi , x j ) i j ⎟
" k ( xni , xn j ) ⎠ Step 2. Compute matrix A with (20).
⎝ ni 1 ni 2
Step 3. Decision is made in favor of the class with the
Following some simple algebraic steps, we see that the first minimum distance in (22).
term in Equation (12) can be reformulated as Output: Label vector for testing data L_test.
T
φq
i
( x) φq
i
( x)
= φ T ( x)φ ( X i ) (φ T ( X i )φ ( X i ) + λ I ) φ T ( X i )
−1

III. EXPERIMENTAL RESULTS

φ ( X i ) (φ ( X i )φ ( X i ) + λ I ) φ ( X i )φ ( x)
−1
T T
(16) In this section, we conduct experiments on synthetic data
sets and UCI data sets to evaluate results of our proposed
= K ( x, X i ) ( K ( X i , X i ) + λ I )
−1

method for classification task and compare its results with

K ( X i , X i ) ( K ( X i , X i ) + λ I ) K ( X i , x)
−1
those of the related classification methods.
= AT K ( X i , X i ) A A. Experimental Setup
Similarly, the second term in Equation (12) can be The proposed KRRC method is compared with the related
reformulated as algorithms, such as KNN, LRC and KLRC. We use Gaussian
kernel in Equation (14) for KRRC and KLRC. In our
experiments, the regularization parameter λ was set as 0.005.
For each data set, we use 5-fold cross-validation to evaluate
the performance of proposed method, i.e., 4 folds are used for

2265
Authorized licensed use limited to: INDIAN INST OF INFO TECH AND MANAGEMENT. Downloaded on December 05,2023 at 12:38:41 UTC from IEEE Xplore. Restrictions apply.
 training and the last fold is used for testing. This process is these two synthetic data sets, while KNN, KLRC and KRRC
repeated 5 times, leaving one different fold for testing each give better results. Since these synthetic data sets has
time. The average accuracy and corresponding standard nonlinear structure and the assumption underlying LRC
deviation over the five runs of cross validation is reported for method is not satisfy.
evaluation.
B. Experiments on Synthetic Data Sets C. Experiments on UCI Data Sets
We first conduct experiments on two synthetic data sets In the experiments, we choose 14 real world data sets with
displayed in Fig. 2 and Fig.3. In these figure, the data points varying dimensions and number of data points from UCI data
that belong to the same class are shown with the same color repository to test our algorithm. The data sets are named as
and style. Obviously, they can’t be classified linearly. The wine, Soybean2, Soybean1, liver, heart, glass, breast, yeast,
performance is shown in Tables I – II. vowel, diabetes, seeds, dermatology, hepatitis and balance
[22]. The detail of the data description is shown in Table III.
Table IV shows the classification results of different methods.
10 The numbers in the brackets are the corresponding standard
deviation. According to Table IV, our method generally
5 shows higher performance than the other methods.
TABLE III
UCI DATA DESCRIPTIONS AND EXPERIMENTAL SETTINGS
0

Data Set #Dimension #Number #Class

-5
wine 13 178 3
Soybean2 35 136 4
-10
Soybean1 35 266 15
liver 6 345 2
heart 13 297 2
-15 glass 9 214 4
breast 9 683 2
yeast 8 1484 10
-20
-10 -8 -6 -4 -2 0 2 4 6 8 10 vowel 10 528 11
diabetes 8 768 2
seeds 7 210 3
Fig.2. The Synthetic Data Set 1
dermatology 34 366 6
hepatitis 19 155 2
TABLE I balance 4 625 3
CLASSIFICATION RESULTS (%) COMPARISONS ON SYNTHETIC DATA SET 1
method KNN LRC KLRC KRRC
Accuracy 88.50 74.00 87.00 89.00 TABLE IV
Standard deviation 5.61 8.46 2.92 3.74 ACCURACY (%) COMPARISONS ON UCI DATA SET

Data Set/Method KNN LRC KLRC KRRC

8
wine 78.05(2.96) 52.92(7.67) 84.87(6.79) 87.14(3.53)
Soybean2 86.75(6.05) 80.16(6.83) 86.75(6.05) 87.49(6.04)
6
Soybean1 86.46(2.51) 85.72(4.36) 89.11(3.63) 89.48(3.49)
4 liver 62.03(5.14) 60.29(5.23) 58.55(4.90) 68.12(3.78)
heart 57.93(2.90) 67.65(4.38) 71.06(2.53) 73.40(4.04)
2 glass 71.99(6.96) 50.96(5.25) 63.62(7.75) 72.43(7.24)
breast 96.19(1.41) 35.43(0.67) 96.93(1.81) 97.36(1.83)
0
yeast 51.75(1.00) 35.92(2.33) 53.10(3.85) 59.84(1.68)
vowel 98.67(0.97) 59.47(2.74) 98.86(0.71) 99.24(0.71)
-2
diabetes 67.71(2.69) 62.10(3.90) 69.01(1.98) 73.18(4.34)
-4
seeds 90.48(2.61) 62.38(4.86) 91.43(2.43) 93.81(3.23)
-3 -2 -1 0 1 2 3
dermatology 88.80(1.57) 91.25(2.55) 93.99(2.22) 94.26(1.03)
Fig.3. The Synthetic Data Set 2 hepatitis 54.19(8.75) 60.65(7.74) 55.48(3.76) 61.29(5.40)
balance 80.15(3.70) 90.72(0.83) 89.92(1.71) 92.79(1.70)

TABLE II However, we cannot conclude which classification method

CLASSIFICATION RESULTS (%) COMPARISONS ON SYNTHETIC DATA SET 2
will certainly beat the others. In this experiment, we see that
method KNN LRC KLRC KRRC
Accuracy 95.50 75.70 96.80 97.00 KRRC performs little better in more of the selected data sets.
Standard deviation 0.84 8.90 0.87 1.05
According to Table I and II, LRC performed not well on

2266
Authorized licensed use limited to: INDIAN INST OF INFO TECH AND MANAGEMENT. Downloaded on December 05,2023 at 12:38:41 UTC from IEEE Xplore. Restrictions apply.
 IV. CONCLUSION [16] Huang, Shih-Ming, and Jar-Ferr Yang. "Improved principal component
regression for face recognition under illumination variations." Signal
In this paper, we presented a kernel ridge regression Processing Letters, IEEE. Vol.19, pp. 179-182. April, 2012.
classification (KRRC) algorithm based on ridge regression [17] Lu, Yuwu, Xiaozhao Fang, and Binglei Xie. (June, 2013) Kernel linear
for classification. KRRC algorithm firstly makes a nonlinear regression for face recognition. Neural Computing and Applications
[Online]. pp. 1-7. Available: http://dx.doi.org/
mapping of the data to a feature space, and then perform 10.1007/s00521-013-1435-6.
ridge regression classification method on this feature space, [18] Hastie, Trevor, et al. The elements of statistical learning: data mining,
so KRRC is good at enhancing the linearity of distribution inference and prediction. The Mathematical Intelligencer 27.2 (2005):
structure underlying samples and able to obtain higher 83-85.
[19] Shawe-Taylor, John, and Nello Cristianini. Kernel methods for pattern
accuracy than LRC. We showed the effective performance of analysis. Cambridge university press, 2004.
our method by comparing its results on the synthetic and UCI [20] Hoerl, Arthur E., and Robert W. Kennard, "Ridge regression: Biased
data sets with related subspace based classification methods. estimation for nonorthogonal problems," Technometrics, Vol. 12, pp.
However, KRRC require matrix inversion computation which 55-67, January, 1970.
[21] Gujarati, Damodar N., and J. B. Madsen, "Basic econometrics,"
can be computationally intensive for high dimensional and Journal of Applied Econometrics. Vol. 13, pp. 209-212, February 1998.
large datasets, including text documents, face images, and [22] Bache, K. & Lichman, M. (2013). UCI Machine Learning Repository.
gene expression data. Therefore, developing efficient Irvine, CA: University of California, School of Information and
algorithms yet with theoretical guarantees will be interesting Computer Science. Available: http://archive.ics.uci.edu/ml
in future research.

ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers
for their constructive comments and suggestions.

REFERENCES
[1] Cover, Thomas, and Peter Hart, "Nearest neighbor pattern
classification," IEEE Transactions on Information Theory, vol.13, pp.
21-27, January 1967.
[2] Fayed, Hatem A., and Amir F. Atiya, "A novel template reduction
approach for the k-nearest neighbor method," IEEE Transactions on
Neural Networks, vol. 20, pp. 890-896, May 2009.
[3] T. Hastie and R. Tibshirani, “Discriminant adaptive nearest neighbor
classification,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 6,
pp. 607–616, Jun. 1996.
[4] J. Peng, D. R. Heisterkamp, and H. K. Dai, “LDA/SVM driven nearest
neighbor classification,” IEEE Trans. Neural Netw., vol. 14, no. 4, pp.
940–942, Jul. 2003.
[5] H. A. Fayed and A. F. Atiya, “A novel template reduction approach for
the K-nearest neighbor method,” IEEE Trans. Neural Netw., vol. 20,
no. 5, pp. 890–896, May 2009.
[6] Y. G. Liu , S. Z. S. Ge , C. G. Li and Z. S. You, "K-NS: A classifier by
the distance to the nearest subspace", IEEE Trans. Neural Netw., vol.
22, no. 8, pp.1256 -1268, 2011.
[7] Oja, Erkki. Subspace methods of pattern recognition. England:
Research Studies Press, 1983, Vol. 4.
[8] Cappelli, Raffaele, Dario Maio, and Davide Maltoni,"Subspace
classification for face recognition," Biometric Authentication. 2002.
[9] Li, Stan Z, "Face recognition based on nearest linear combinations,"
IEEE Computer Society Conference on Computer Vision and Pattern
Recognition, IEEE, 1998.
[10] Li, Stan Z., and Juwei Lu, "Face recognition using the nearest feature
line method," IEEE Transactions on Neural Networks, Vol. 10, pp.
439-443, February, 1999.
[11] S. Watanabe, P. F. Lambert, C. A. Kulikowski, J. L. Buxton, and R.
Walker, Evaluation and selection of variables in pattern recognition, In
Computer and Information Sciences II. New York: Academic, 1967.
[12] Naseem, Imran, Roberto Togneri, and Mohammed Bennamoun,
"Linear regression for face recognition," IEEE Trans. Pattern Analysis
and Machine Intelligence, vol.32, pp.2106-2112, 2010.
[13] Huber, Peter J. Robust statistics. Springer Berlin Heidelberg, 2011.
[14] Ren, Chuan-Xian, Dao-Qing Dai, and Hong Yan, "L2,1-norm based
Regression for Classification," 2011 First Asian Conference on Pattern
Recognition (ACPR), IEEE, 2011.
[15] Naseem, Imran, Roberto Togneri, and Mohammed Bennamoun,
"Robust regression for face recognition," Pattern Recognition, vol.45,
pp. 104-118, January 2012.

2267
Authorized licensed use limited to: INDIAN INST OF INFO TECH AND MANAGEMENT. Downloaded on December 05,2023 at 12:38:41 UTC from IEEE Xplore. Restrictions apply.