Abstract
Support vector machines regression (SVMR) is a regularized learning algorithm in reproducing kernel Hilbert spaces with a loss function called the ε-insensitive loss function. Compared with the well-understood least square regression, the study of SVMR is not satisfactory, especially the quantitative estimates of the convergence of this algorithm. This paper provides an error analysis for SVMR, and introduces some recently developed methods for analysis of classification algorithms such as the projection operator and the iteration technique. The main result is an explicit learning rate for the SVMR algorithm under some assumptions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
N. Aronszajn, Theory of reproducing kernels, Trans. Am. Math. Soc. 68, 337–404 (1950).
P.L. Bartlett, The sample complexity of pattern classification with neural networks: The size of the weights is more import than the size of the network, IEEE Trans. Inf. Theory 44, 525–536 (1998).
O. Bousquet, A. Elisseeff, Stability and generalization, J. Mach. Learn. Res. 2, 499–526 (2002).
D.R. Chen, Q. Wu, Y. Ying, D.X. Zhou, Support vector machine soft margin classifiers: error analysis, J. Mach. Learn. Res. 5, 1143–1175 (2004).
A. Christmann, I. Steinwart, Consistency and robustness of kernel-based regression in convex risk minimization, Bernoulli 13, 799–819 (2007).
F. Cucker, S. Smale, On the mathematical foundations of learning theory, Bull. Am. Math. Soc. 39, 1–49 (2001).
F. Cucker, S. Smale, Best choices for regularization parameters in learning theory: On the bias-variance problem, Found. Comput. Math. 2, 413–428 (2002).
E. De Vito, A. Caponnetto, L. Rosasco, Model selection for regularized least-squares algorithm in learning theory, Found. Comput. Math. 5, 59–85 (2005).
L. Devroye, L. Györfi, G. Lugosi, A Probabilistic Theory of Pattern Recognition (Springer, New York, 1997).
T. Evgeniou, M. Pontil, T. Poggio, Regularization networks and support vector machines, Adv. Comput. Math. 13, 1–50 (2000).
P.J. Huber, Robust Statistics (Wiley, New York, 1981).
T. Poggio, S. Smale, The mathematics of learning: Deal with data, Not. Am. Math. Soc. 50, 537–544 (2003).
M. Pontil, S. Mukherjee, F. Girosi, On the noise model of support vector machine regression, A.I. Memo 1651, MIT Artificial Intelligence Lab., 1998.
L. Rosasco, E. De Vito, A. Caponnetto, M. Piana, A. Verri, Are loss functions all the same? Neural Comput. 16, 1063–1076 (2004).
B. Scholkopf, A.J. Smola, Learning with Kernel (MIT Press, Cambridge, 2002).
S. Smale, D.X. Zhou, Shannon sampling II. Connections to learning theory, Appl. Comput. Harmon. Anal. 19, 285–302 (2005).
S. Smale, D.X. Zhou, Learning theory estimates via integral operators and their applications, Constr. Approx. 26, 153–172 (2007).
C. Scovel, I. Steinwart, Fast rates for support vector machine, in Proceedings of the Conference on Learning Theory (COLT-2005), pp. 279–294.
V. Vapnik, The Nature of Statistical Learning Theory (Springer, New York, 1995).
V. Vapnik, Statistical Learning Theory (Wiley, New York, 1998).
Q. Wu, D.X. Zhou, SVM soft margin classifiers: Linear programming versus quadratic programming, Neural Comput. 17, 1160–1187 (2005).
Q. Wu, Y. Ying, D.X. Zhou, Learning rates of least-square regularized regression, Found. Comput. Math. 6, 171–192 (2006).
D.X. Zhou, The covering number in learning theory, J. Complex. 18, 739–767 (2002).
D.X. Zhou, Capacity of reproducing kernel spaces in learning theory, IEEE Trans. Inf. Theory 49, 1743–1752 (2003).
D.X. Zhou, K. Jetter, Approximation with polynomial kernels and SVM classifiers, Adv. Comput. Math. 25, 323–344 (2006).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Felipe Cucker.
Research supported by NNSF of China No. 10471002, No. 10571010 and RFDP of China No. 20060001010.
Rights and permissions
About this article
Cite this article
Tong, H., Chen, DR. & Peng, L. Analysis of Support Vector Machines Regression. Found Comput Math 9, 243–257 (2009). https://doi.org/10.1007/s10208-008-9026-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10208-008-9026-0
Keywords
- Support vector machines regression
- Regularization
- Learning rates
- Reproducing kernel Hilbert spaces
- Excess error