Article

A kernel path algorithm for support vector machines

Authors:

Frederick H. LochovskyAuthors Info & Claims

ICML '07: Proceedings of the 24th international conference on Machine learning

Pages 951 - 958

https://doi.org/10.1145/1273496.1273616

Published: 20 June 2007 Publication History

Abstract

The choice of the kernel function which determines the mapping between the input space and the feature space is of crucial importance to kernel methods. The past few years have seen many efforts in learning either the kernel function or the kernel matrix. In this paper, we address this model selection issue by learning the hyperparameter of the kernel function for a support vector machine (SVM). We trace the solution path with respect to the kernel hyperparameter without having to train the model multiple times. Given a kernel hyperparameter value and the optimal solution obtained for that value, we find that the solutions of the neighborhood hyperparameters can be calculated exactly. However, the solution path does not exhibit piecewise linearity and extends nonlinearly. As a result, the breakpoints cannot be computed in advance. We propose a method to approximate the breakpoints. Our method is both efficient and general in the sense that it can be applied to many kernel functions in common use.

References

[1]

Bach, F., Lanckriet, R., & Jordan:, M. (2004). Multiple kernel learning, conic duality, and the SMO algorithm. Proceedings of the 21th International Conference on Machine Learning (ICML-04).

Digital Library

[2]

Bach, F., Thibaux, R., & Jordan, M. (2005). Regularization paths for learning multiple kernels. Advances in Neural Information Processing Systems 17 (NIPS-05).

[3]

Chang, C., & Lin, C. (2001). LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm/.

[4]

Cristianini, N., Kandola, J., Elissee, A., & Shawe-Taylor, J. (2002). On kernel target alignment. Advances in Neural Information Processing Systems 15 (NIPS-02).

[5]

Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32, 407--499.

[6]

Hastie, T., Rosset, S., Tibshirani, R., & Zhu, J. (2004). The entire regularization path for the support vector machine. Journal of Machine Learning Research, 5, 1391--1415.

Digital Library

[7]

Keerthi, S., Sindhwani, V., & Chapelle, O. (2006). An efficient method for gradient-based adaption of hyperparameters in SVM models. Advances in Neural Information Processing Systems 19 (NIPS-06).

[8]

Lanckriet, G., Cristianini, N., Bartlett, P., Ghaoui, L., & Jordan, M. (2004). Learning the kernel matrix with semidefinite programming. Journal of Machine Learning Research, 5, 27--72.

Digital Library

[9]

Müller, K., Mika, S., Rätsch, G., Tsuda, K., & Schöölkopf, B. (2001). An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 12, 181--202.

Digital Library

[10]

Ong, C., Smola, A., & Williamson, R. (2005). Learning the kernel with hyperkernels. Journal of Machine Learning Research, 6, 1043--1071.

Digital Library

[11]

Rosset, S. (2004). Following curved regularized optimization solution paths. Advances in Neural Information Processing Systems 17 (NIPS-04).

[12]

Rosset, S., & Zhu, J. (2003). Piecewise linear regularized solution paths (Technical Report). Stanford University.

[13]

Schölkopf, B., & Smola, A. (2002). Learning with kernels. MIT Press.

[14]

Sonnenburg, S., Rätsch, G., Schäfer, C., & Schöölkopf, B. (2006). Large scale multiple kernel learning. Journal of Machine Learning Research, 7, 15311565.

Digital Library

[15]

Wang, G., Yeung, D., & Lochovsky, F. (2006). Two-dimensional solution path for support vector regression. Proceedings of the 23th International Conference on Machine Learning (ICML-06).

Digital Library

[16]

Zhang, Z., Kwok, J., & Yeung, D. (2006). Model-based transductive learning of the kernel matrix. Machine Learning, 63, 69--101.

Digital Library

[17]

Zhu, J., Rosset, S., Hastie, T., & Tibshirani, R. (2003). 1-norm support vector machines. Advances in Neural Information Processing Systems 16 (NIPS-03).

Cited By

El Moutaouakil KRoudani MOuhmid AZhilenkov AMobayen S(2024)Decomposition and Symmetric Kernel Deep Neural Network Fuzzy Support Vector MachineSymmetry10.3390/sym1612158516:12(1585)Online publication date: 27-Nov-2024
https://doi.org/10.3390/sym16121585
Zhai ZHuang HGu B(2024)Kernel Path for Semisupervised Support Vector MachineIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.318382535:2(1512-1522)Online publication date: Feb-2024
https://doi.org/10.1109/TNNLS.2022.3183825
Xiong ZLing CGu B(2023)Kernel Error Path AlgorithmIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.315395334:11(8866-8878)Online publication date: Nov-2023
https://doi.org/10.1109/TNNLS.2022.3153953
Show More Cited By

A kernel path algorithm for support vector machines
1. Computing methodologies

Recommendations

Optimizing the kernel selection for support vector machines using performance measures
A2CWiC '10: Proceedings of the 1st Amrita ACM-W Celebration on Women in Computing in India

Support Vector Machine (SVM) is a powerful classification technique based on the idea of Structural Risk Minimization. The main idea behind the Support Vector Machine is to separate the classes with a surface that maximizes the margin between them. Key ...
Wavelet twin support vector machines based on glowworm swarm optimization

Twin support vector machine is a machine learning algorithm developing from standard support vector machine. The performance of twin support vector machine is always better than support vector machine on datasets that have cross regions. Recently ...
Autocorrelation Kernel Functions for Support Vector Machines
ICNC '07: Proceedings of the Third International Conference on Natural Computation - Volume 01

Kernel functions (kernel) are key part and the hard issue of Support Vector Machines. We research the relation of kernel functions and nonlinear mappings and mapped spaces. A new kind of admissible Support Vector Machines kernel is presented. It is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '07: Proceedings of the 24th international conference on Machine learning

June 2007

1233 pages

ISBN:9781595937933

DOI:10.1145/1273496

Editor:
Zoubin Ghahramani
University of Cambridge, United Kingdom

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Machine Learning Journal

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

ICML '07 & ILP '07

Sponsor:

ICML '07 & ILP '07: The 24th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

June 20 - 24, 2007

Oregon, Corvalis, USA

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
361
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

El Moutaouakil KRoudani MOuhmid AZhilenkov AMobayen S(2024)Decomposition and Symmetric Kernel Deep Neural Network Fuzzy Support Vector MachineSymmetry10.3390/sym1612158516:12(1585)Online publication date: 27-Nov-2024
https://doi.org/10.3390/sym16121585
Zhai ZHuang HGu B(2024)Kernel Path for Semisupervised Support Vector MachineIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.318382535:2(1512-1522)Online publication date: Feb-2024
https://doi.org/10.1109/TNNLS.2022.3183825
Xiong ZLing CGu B(2023)Kernel Error Path AlgorithmIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.315395334:11(8866-8878)Online publication date: Nov-2023
https://doi.org/10.1109/TNNLS.2022.3153953
Gu BXiong ZLi XZhai ZZheng G(2023)Kernel Path for ν-Support Vector ClassificationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.309724834:1(490-501)Online publication date: Jan-2023
https://doi.org/10.1109/TNNLS.2021.3097248
Tang GFan N(2022)A Survey of Solution Path Algorithms for Regression and Classification ModelsAnnals of Data Science10.1007/s40745-022-00386-99:4(749-789)Online publication date: 25-Mar-2022
https://doi.org/10.1007/s40745-022-00386-9
Meena GBabita Mohbey K(2022)Assessment of Network Intrusion Detection System Based on Shallow and Deep Learning ApproachesEmerging Technologies in Computer Engineering: Cognitive Computing and Intelligent IoT10.1007/978-3-031-07012-9_28(310-335)Online publication date: 26-May-2022
https://doi.org/10.1007/978-3-031-07012-9_28
Gu BLing C(2021)Generalized error path algorithmPattern Recognition10.1016/j.patcog.2021.108112120:COnline publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1016/j.patcog.2021.108112
Hu LChen YHildebrandt MCastillo CCelis ERuggieri STaylor LZanfir-Fortuna G(2020)Fair classification and social welfareProceedings of the 2020 Conference on Fairness, Accountability, and Transparency10.1145/3351095.3372857(535-545)Online publication date: 27-Jan-2020
https://dl.acm.org/doi/10.1145/3351095.3372857
Gu BSheng VTay KRomano WLi S(2015)Incremental Support Vector Learning for Ordinal RegressionIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2014.234253326:7(1403-1416)Online publication date: Jul-2015
https://doi.org/10.1109/TNNLS.2014.2342533
Le Thi HNguyen M(2014)Self-organizing maps by difference of convex functions optimizationData Mining and Knowledge Discovery10.1007/s10618-014-0369-728:5-6(1336-1365)Online publication date: 1-Sep-2014
https://dl.acm.org/doi/10.1007/s10618-014-0369-7
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten