$\ell _1$ Regularized Robust and Sparse Linear Modeling Using Discrete Optimization

Abstract

In regression, feature selection is an effective strategy to handle contaminated data and to deal with high dimensionality while providing better prediction. In addition to the presence of spurious variables, estimators suffer form corrupted, incorrectly measured or misrecorded observations known as outliers. The natural way to select relevant variables and to detect outliers is done by using the $\ell _0$ norm for both aspects and recast the obtained optimization problem as a mixed integer optimization (MIO) problem. The $\ell _0$ norm estimators perform well when the signal to noise ratio (SNR) is high. However, its performance decreases when the SNR is low due to the overfitting behavior of the $\ell _{0}$ norm when the noise is relatively high. To fix this problem, we propose to regularize the $\ell _{0}$ norm problem for variable selection and outlier detection by adding an $\ell _1$ penalty term. We also propose an efficient and scalable non-convex proximal alternate algorithm producing high quality solution in a short time and used as a warm start for the MIO solver. An empirical comparison between the $\ell _0$ norm approach and its $\ell _1$ regularized extension is presented as well. Results provided that the MIO regularized approach and its discrete first order warm start provide high quality solutions and performs better then the $\ell _{0}$ approach especially for low SNR values.