Abstract
In machine learning problems in general, and in classification in particular, overfitting and inaccuracies can be obtained because of the presence of spurious features and outliers. Unfortunately, this is a frequent situation when dealing with real data. To handle outliers proneness and achieve variable selection, we propose a robust method performing the outright rejection of discordant observations together with the selection of relevant variables. A natural way to define the corresponding optimization problem is to use the \(\ell _0\) norm and recast it as a mixed integer optimization problem (MIO) having a unique global solution, benefiting from algorithmic advances in integer optimization combined with hardware improvements. We also present an empirical comparison between the \(\ell _0\) norm approach, the 0–1 loss and the hinge loss classification problems. Results on both synthetic and real data sets showed that, the proposed approach provides high quality solutions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)
Lee, Y.: Support vector machines for classification: a statistical portrait. In: Bang, H., Zhou, X., van Epps, H., Mazumdar, M. (eds.) Statistical Methods in Molecular Biology. Methods in Molecular Biology (Methods and Protocols), vol. 620, pp. 347–368. Humana Press, Totowa (2010). https://doi.org/10.1007/978-1-60761-580-4_11
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. SSS. Springer, New York (2009). https://doi.org/10.1007/978-0-387-84858-7
Bertsimas, D., King, A., Mazumder, R., et al.: Best subset selection via a modern optimization lens. Ann. Stat. 44(2), 813–852 (2016)
Tang, Y., Li, X., Xu, Y., Liu, S., Ouyang, S.: A mixed integer programming approach to maximum margin 0–1 loss classification. In: 2014 International Radar Conference, pp. 1–6. IEEE (2014)
Liu, H., Motoda, H.: Computational Methods of Feature Selection. CRC Press, Chapman (2007)
Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., Vapnik, V.: Feature selection for SVMs. In: Advances in Neural Information Processing Systems, pp. 668–674 (2001)
Frohlich, H., Chapelle, O., Scholkopf, B.: Feature selection for support vector machines by means of genetic algorithm. In: Proceedings of the 15th IEEE International Conference on Tools with Artificial Intelligence, pp. 142–148. IEEE (2003)
Minh Hoai Nguyen and Fernando De la Torre: Optimal feature selection for support vector machines. Pattern Recogn. 43(3), 584–591 (2010)
Liu, Z., Elashoff, D., Piantadosi, S.: Sparse support vector machines with L0 approximation for ultra-high dimensional omics data. Artif. Intell. Med. 96, 134–141 (2019)
Frénay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE Trans. Neural Netw. Learn. Syst. 25(5), 845–869 (2013)
Song, Q., Wenjie, H., Xie, W.: Robust support vector machine with bullet hole image classification. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 32(4), 440–448 (2002)
Yichao, W., Liu, Y.: Adaptively weighted large margin classifiers. J. Comput. Graph. Stat. 22(2), 416–432 (2013)
Lin, C.-F., Wang, S.-D.: Fuzzy support vector machines. IEEE Trans. Neural Networks 13(2), 464–471 (2002)
Batuwita, R., Palade, V.: FSVM-CIL: fuzzy support vector machines for class imbalance learning. IEEE Trans. Fuzzy Syst. 18(3), 558–571 (2010)
Ding, H., Xu, J.: Random gradient descent tree: a combinatorial approach for SVM with outliers. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)
Guibiao, X., Cao, Z., Bao-Gang, H., Principe, J.C.: Robust support vector machines based on the rescaled hinge loss function. Pattern Recogn. 63, 139–148 (2017)
Xiao, Y., Wang, H., Wenli, X.: Ramp loss based robust one-class SVM. Pattern Recogn. Lett. 85, 15–20 (2017)
Yang, L., Dong, H.: Robust support vector machine with generalized quantile loss for classification and regression. Appl. Soft Comput. 81, 105483 (2019)
Bertsimas, D., Dunn, J., Pawlowski, C., Zhuo, Y.D.: Robust classification. INFORMS J. Optim. 1(1), 2–34 (2018)
Chen, L., Zhou, S.: Sparse algorithm for robust LSSVM in primal space. Neurocomputing 275, 2880–2891 (2018)
Wang, K., Zhong, P.: Robust non-convex least squares loss function for regression with outliers. Knowl. Based Syst. 71, 290–302 (2014)
Yang, X., Tan, L., He, L.: A robust least squares support vector machine for regression and classification with noise. Neurocomputing 140, 41–52 (2014)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Jammal, M., Canu, S., Abdallah, M. (2020). Robust and Sparse Support Vector Machines via Mixed Integer Programming. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2020. Lecture Notes in Computer Science(), vol 12566. Springer, Cham. https://doi.org/10.1007/978-3-030-64580-9_47
Download citation
DOI: https://doi.org/10.1007/978-3-030-64580-9_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64579-3
Online ISBN: 978-3-030-64580-9
eBook Packages: Computer ScienceComputer Science (R0)