Summary
For the recent NIPS-2003 feature selection challenge we studied ensembles of regularized least squares classifiers (RLSC). We showed that stochastic ensembles of simple least squares kernel classifiers give the same level of accuracy as the best single RLSC. Results achieved were ranked among the best at the challenge. We also showed that performance of a single RLSC is much more sensitive to the choice of kernel width than that of an ensemble. As a continuation of this work we demonstrate that stochastic ensembles of least squares classifiers with randomized kernel widths and OOB-post-processing often outperform the best single RLSC, and require practically no parameter tuning. We used the same set of very high dimensional classification problems presented at the NIPS challenge. Fast exploratory Random Forests were applied for variable filtering first.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
A. Borisov, V. Eruhimov, and E. Tuv. Dynamic soft feature selection for tree-based ensembles. In I. Guyon, S. Gunn, M. Nikravesh, and L. Zadeh, editors, Feature Extraction, Foundations and Applications. Springer, Berlin Heidelberg New York, 2005
O. Bousquet and A. Elisseeff. Algorithmic stability and generalization performance. In NIPS, pages 196–202, 2000
L. Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996
L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001
L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone. Classification and Regression Trees. CRC, Boca Raton, FL, 1984
F. Cucker and S. Smale. On the mathematial foundations of learning. Bulletin of the American Mathematical Society, 89(1):1–49, 2001
F. Cucker and S. Smale. Best choices for regularization parameters in learning theory: on the bias-variance problem. Foundations of Computational Mathematics, 2(4):413–428, 2003
T. Evgeniou. Learning with Kernel Machine Architectures. PhD thesis, Massachusetts Institute of Technology, EECS, July 2000
Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. In European Conference on Computational Learning Theory, pages 23–37, 1995
J.H. Friedman. Greedy function approximation: a gradient boosting machine. Technical report, Department of Statistics, Stanford University, 1999
J.H. Friedman. Stochastic gradient boosting. Technical report, Department of Statistics, Stanford University, 1999
A. Hoerl and R. Kennard. Ridge regression; biased estimation for nonorthogonal problems. Technometrics, 12(3):55–67, 1970
T. Poggio, R. Rifkin, S. Mukherjee, and A. Rakhlin. Bagging regularizes. CBCL Paper 214, Massachusetts Institute of Technology, Cambridge, MA, February 2002. AI Memo #2002–2003
T. Poggio and S. Smale. The mathematics of learning: Dealing with data. Notices of the American Mathematical Society (AMS), 50(5):537–544, 2003
R. Rifkin. Everything Old is New Again: A Fresh Look at Historical Approaches in Machine Learning. PhD thesis, MIT, 2002
J.A.K. Suykens and J. Vandervalle. Least squares support vector machines. Neural Processing Letters, 9(3):293–300, June 1999
A.N. Tikhonov and V.Y. Arsenin. Solutions of Ill-posed Problems. W.H. Wingston, Washington DC, 1977
K. Torkkola. Feature extraction by non-parametric mutual information maximization. Journal of Machine Learning Research, 3:1415–1438, March 2003
K. Torkkola and E. Tuv. Ensembles of regularized least squares classifiers for high-dimensional problems. In I. Guyon, S. Gunn, M. Nikravesh, and L. Zadeh, editors, Feature Extraction, Foundations and Applications. Springer, Berlin Heidelberg New York, 2005
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Torkkola, K., Tuv, E. (2008). Ensembles of Least Squares Classifiers with Randomized Kernels. In: Lin, T.Y., Xie, Y., Wasilewska, A., Liau, CJ. (eds) Data Mining: Foundations and Practice. Studies in Computational Intelligence, vol 118. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-78488-3_22
Download citation
DOI: https://doi.org/10.1007/978-3-540-78488-3_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-78487-6
Online ISBN: 978-3-540-78488-3
eBook Packages: EngineeringEngineering (R0)