article

Free access

SVMTorch: support vector machines for large-scale regression problems

Authors:

Ronan Collobert,

Samy BengioAuthors Info & Claims

The Journal of Machine Learning Research, Volume 1

Pages 143 - 160

https://doi.org/10.1162/15324430152733142

Published: 01 September 2001 Publication History

Abstract

Support Vector Machines (SVMs) for regression problems are trained by solving a quadratic optimization problem which needs on the order of l square memory and time resources to solve, where l is the number of training examples. In this paper, we propose a decomposition algorithm, SVMTorch (available at <a target=_new href=http://www.idiap.ch/learning/SVMTorch.html>http://www.idiap.ch/learning/SVMTorch.html</a>), which is similar to SVM-Light proposed by Joachims (1999) for classification problems, but adapted to regression problems. With this algorithm, one can now efficiently solve large-scale regression problems (more than 20000 examples). Comparisons with Nodelib, another publicly available SVM algorithm for large-scale regression problems from Flake and Lawrence (2000) yielded significant time improvements. Finally, based on a recent paper from Lin (2000), we show that a convergence proof exists for our algorithm.

References

[1]

Collobert, R., & Bengio, S. (2000). On the Convergence of SVMTorch, an Algorithm for Large-Scale Regression Problems (IDIAP-RR No. 24). IDIAP. (Available at ftp://www.idiap.ch/pub/reports/2000/rr00-24.ps.gz)

[2]

Drucker, H., Burges, C., Kaufman, L., Smola, A., & Vapnik, V. (1997). Support vector regression machines. In M. Mozer, M. Jordan, & T. Petsche (Eds.), Advances in Neural Information Processing Systems 9 (pp. 155-161). The MIT Press.

[3]

Flake, G., & Lawrence, S. (2000). Efficient SVM regression training with SMO. (Submitted to Machine Learning. Available at http://external.nj.nec.com/homepages/flake/smorch.ps)

[4]

Fletcher, R. (1987). Practical Methods of Optimization. Chichester: John Wiley and Sons.

[5]

Joachims, T. (1999). Making large-scale support vector machine learning practical. In B. Schölkopf, C. Burges, & A. Smola (Eds.), Advances in Kernel Methods. The MIT Press.

[6]

Keerthi, S. S., & Gilbert, E. G. (2000). Convergence of a Generalized SMO Algorithm for SVM Classifier Design (Tech. Rep. No. CD-00-01). Control Division, Dept. of Mechanical and Production Engineering, National University of Singapore. (Available at http://guppy.mpe.nus.edu.sg/~mpessk/svm/conv_ml.ps.gz)

[7]

Keerthi, S. S., Shevade, S. K., Bhattacharyya, C., & Murthy, K. R. K. (1999). Improvements to Platt's SMO Algorithm for SVM Classifier Design (Tech. Rep. No. CD-99-14). Control Division, Dept. of Mechanical and Production Engineering, National University of Singapore. (To appear in Neural Computation. Available at http://guppy.mpe.nus.edu.sg/~mpessk/smo_mod.ps.gz)

[8]

Laskov, P. (2000). An improved decomposition algorithm for regression support vector machines. In S. Solla, T. Leen, & K.-R. Müller (Eds.), Advances in Neural Information Processing Systems 12. The MIT Press.

[9]

Lin, C. (2000). On the Convergence of the Decomposition Method for Support Vector Machines (Tech. Rep.). National Taiwan University. (Available at http://www.csie.ntu.edu.tw/~cjlin/papers/conv.ps.gz)

[10]

Müller, K.-R., Smola, A., Rätsch, G., Schölkopf, B., Kohlmorgen, J., & Vapnik, V. (1997). Predicting time series with support vector machines. In W. Gerstner, A. Germond, M. Hasler, & J.-D. Nicoud (Eds.), Artificial Neural Networks - ICANN'97 (pp. 999- 1004). Springer.

[11]

Osuna, E., Freund, R., & Girosi, F. (1997). An improved training algorithm for support vector machines. In J. Principe, L. Giles, N. Morgan, & E. Wilson (Eds.), Neural Networks for Signal Processing VII - Proceedings of the 1997 IEEE Workshop (pp. 276-285). New York: IEEE Press.

[12]

Platt, J. C. (1999). Fast training of support vector machines using sequential minimal optimization. In B. Schölkopf, C. Burges, & A. Smola (Eds.), Advances in Kernel Methods. The MIT Press.

Digital Library

[13]

Shevade, S. K., Keerthi, S. S., Bhattacharyya, C., & Murthy, K. R. K. (2000). Improvements to the SMO algorithm for SVM regression. IEEE Transaction on Neural Networks, 11 (5), 1188-1183.

Digital Library

[14]

Smola, A., & Schölkopf, B. (1998). A Tutorial on Support Vector Regression (Tech. Rep. No. NeuroCOLT NC-TR-98-030). Royal Holloway College, University of London, UK.

[15]

Vapnik, V. (1995). The Nature of Statistical Learning Theory (second ed.). Springer.

Cited By

Leiva LVázquez MTorrents-Barrena J(2022)FPGA acceleration analysis of LibSVM predictors based on high-level synthesisThe Journal of Supercomputing10.1007/s11227-022-04406-678:12(14137-14163)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s11227-022-04406-6
Pham QKumar MDi Nunno FElbeltagi AGranata FIslam ATalukdar SNguyen XAhmed AAnh D(2022)Groundwater level prediction using machine learning algorithms in a drought-prone areaNeural Computing and Applications10.1007/s00521-022-07009-734:13(10751-10773)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s00521-022-07009-7
Bhatt CKumar IVijayakumar VSingh KKumar A(2021)The state of the art of deep learning models in medical science and their challengesMultimedia Systems10.1007/s00530-020-00694-127:4(599-613)Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s00530-020-00694-1
Show More Cited By

Recommendations

Wavelet twin support vector machines based on glowworm swarm optimization

Twin support vector machine is a machine learning algorithm developing from standard support vector machine. The performance of twin support vector machine is always better than support vector machine on datasets that have cross regions. Recently ...
Insensitive stochastic gradient twin support vector machines for large scale problems
Abstract
Within the large scale classification problem, the stochastic gradient descent method called PEGASOS has been successfully applied to support vector machines (SVMs). In this paper, we propose a stochastic gradient twin support vector ...
PAC-Bayes bounds for twin support vector machines

Twin support vector machines are regarded as a milestone in the development of support vector machines. Compared to standard support vector machines, they learn two nonparallel hyperplanes rather than one as in standard support vector machines for ...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 1, Issue

9/1/2001

348 pages

ISSN:1532-4435

EISSN:1533-7928

Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 September 2001

Published in JMLR Volume 1

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

157
Total Citations
View Citations
1,953
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)17

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Leiva LVázquez MTorrents-Barrena J(2022)FPGA acceleration analysis of LibSVM predictors based on high-level synthesisThe Journal of Supercomputing10.1007/s11227-022-04406-678:12(14137-14163)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s11227-022-04406-6
Pham QKumar MDi Nunno FElbeltagi AGranata FIslam ATalukdar SNguyen XAhmed AAnh D(2022)Groundwater level prediction using machine learning algorithms in a drought-prone areaNeural Computing and Applications10.1007/s00521-022-07009-734:13(10751-10773)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s00521-022-07009-7
Bhatt CKumar IVijayakumar VSingh KKumar A(2021)The state of the art of deep learning models in medical science and their challengesMultimedia Systems10.1007/s00530-020-00694-127:4(599-613)Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s00530-020-00694-1
Pandove DGoel SRani R(2019)General correlation coefficient based agglomerative clusteringCluster Computing10.1007/s10586-018-2863-y22:2(553-583)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s10586-018-2863-y
Gupta UGupta D(2019)An improved regularization based Lagrangian asymmetric ν-twin support vector regression using pinball loss functionApplied Intelligence10.1007/s10489-019-01465-w49:10(3606-3627)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s10489-019-01465-w
Nayak RMishra DRath A(2019)An optimized SVM-k-NN currency exchange forecasting model for Indian currency marketNeural Computing and Applications10.1007/s00521-017-3248-531:7(2995-3021)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1007/s00521-017-3248-5
Fox SBoland DLeong PAnderson JBazargan K(2018)FPGA Fastfood - A High Speed Systolic Implementation of a Large Scale Online Kernel MethodProceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays10.1145/3174243.3174271(279-284)Online publication date: 15-Feb-2018
https://dl.acm.org/doi/10.1145/3174243.3174271
Pandove DGoel SRani R(2018)Systematic Review of Clustering High-Dimensional and Large DatasetsACM Transactions on Knowledge Discovery from Data10.1145/313208812:2(1-68)Online publication date: 23-Jan-2018
https://dl.acm.org/doi/10.1145/3132088
Hu LHu JYe ZShen CPeng Y(2018)Performance Analysis for SVM Combining with Metric LearningNeural Processing Letters10.1007/s11063-017-9771-748:3(1373-1394)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1007/s11063-017-9771-7
Yu HLu JZhang G(2018)An Incremental Dual nu-Support Vector Regression AlgorithmAdvances in Knowledge Discovery and Data Mining10.1007/978-3-319-93037-4_41(522-533)Online publication date: 3-Jun-2018
https://dl.acm.org/doi/10.1007/978-3-319-93037-4_41
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents