Abstract
In this paper, we investigate a class of heuristic schemes to solve the NP-hard problem of minimizing \(\ell _0\)-norm over a convex set. A well-known approximation is to consider the convex problem of minimizing \(\ell _1\)-norm. We are interested in finding improved results in cases where the problem in \(\ell _1\)-norm does not provide an optimal solution to the \(\ell _0\)-norm problem. We consider a relaxation technique using a family of smooth concave functions depending on a parameter. Some other relaxations have already been tried in the literature and the aim of this paper is to provide a more general context. This motivation allows deriving new theoretical results that are valid for general constraint set. We use a homotopy algorithm, starting from a solution to the problem in \(\ell _1\)-norm and ending in a solution of the problem in \(\ell _0\)-norm. The new results are existence of the solutions of the subproblem, convergence of the scheme, a monotonicity of the solutions and an exact penalization theorem independent of the data.
Similar content being viewed by others
References
Abdallah, L., Haddou, M., Migot, T.: Solving absolute value equation using complementarity and smoothing functions. J. Comput. Appl. Math. 327, 196–207 (2018)
Asif, M.S., Romberg, J.: Sparse recovery of streaming signals using \(\ell _1 \)-homotopy. IEEE Trans. Signal Process. 62(16), 4209–4223 (2014)
Auslender, A., Teboulle, M.: Asymptotic Cones and Functions in Optimization and Variational Inequalities. Springer, Berlin (2006)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)
Bergounioux, M., Haddou, M.: A new relaxation method for a discrete image restoration problem. J. Convex Anal. 17(3), 421–435 (2010)
Bi, S., Liu, X., Pan, S.: Exact penalty decomposition method for zero-norm minimization based on mpec formulation. SIAM J. Sci. Comput. 36(4), A1451–A1477 (2014)
Blumensath, T., Davies, M.E.: Gradient pursuits. IEEE Trans. Signal Process. 56(6), 2370–2382 (2008)
Bradley, P.S., Mangasarian, O.L.: Feature selection via concave minimization and support vector machines. ICML 98, 82–90 (1998)
Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009)
Candes, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)
Candès, E.J., Romberg, J.K., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)
Candes, E.J., Romberg, J.K., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)
Candes, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)
Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\) minimization. J. Fourier Anal. Appl. 14(5), 877–905 (2008)
Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14(10), 707–710 (2007)
Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)
Chen, X., Zhou, W.: Convergence of the reweighted \(\ell _1\) minimization algorithm for \(\ell _2-\ell _p\) minimization. Comput. Optim. Appl. 59(1–2), 47–61 (2014)
Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and best \(k\)-term approximation. J. Am. Math. Soc. 22(1), 211–231 (2009)
Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)
Di Lorenzo, D., Liuzzi, G., Rinaldi, F., Schoen, F., Sciandrone, M.: A concave optimization-based approach for sparse portfolio selection. Optim. Methods Softw. 27(6), 983–1000 (2012)
Donoho, D. L.: Neighborly polytopes and sparse solutions of underdetermined linear equations. Technical report, Department of Statistics, Stanford University (2004)
Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)
Donoho, D.L.: For most large underdetermined systems of linear equations the minimal \(\ell _1\)-norm solution is also the sparsest solution. Commun. Pure Appl. Math. 59(6), 797–829 (2006)
Donoho, D.L., Elad, M.: Optimally sparse representation in general (nonorthogonal) dictionaries via \(\ell _1\) minimization. Proc. Natl. Acad. Sci. 100(5), 2197–2202 (2003)
Donoho, D.L., Elad, M., Temlyakov, V.N.: Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inf. Theory 52(1), 6–18 (2006)
Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory 47(7), 2845–2862 (2001)
Feng, M., Mitchell, J.E., Pang, J.-S., Shen, X., Wächter, A.: Complementarity formulations of l0-norm optimization problems. Technical report, Technical report, Department of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY (2013)
Foucart, S., Lai, M.-J.: Sparsest solutions of underdetermined linear systems via \(\ell _q\)-minimization for \(0< q \le 1\). Appl. Comput. Harmonic Anal. 26(3), 395–407 (2009)
Friedman, J., Hastie, T., Tibshirani, R.: The elements of statistical learning. Springer series in statistics, vol. 1. Springer, New York (2001)
Fuchs, J.-J.: On sparse representations in arbitrary redundant bases. IEEE Trans. Inf. Theory 50(6), 1341–1344 (2004)
Fung, G.M., Mangasarian, O.L.: Equivalence of minimal \(\ell _0\) and \(\ell _p\) norm solutions of linear equalities, inequalities and linear programs for sufficiently small p. J. Optim. Theory Appl. 151(1), 1–10 (2011)
Ge, D., Jiang, X., Ye, Y.: A note on the complexity of l p minimization. Math. Program. 129(2), 285–299 (2011)
Gribonval, R., Nielsen, M.: Sparse decomposition in unions of bases. IEEE Trans. Inf. Theory 49(12), 3320–3325 (2003)
Haddou, M., Maheux, P.: Smoothing methods for nonlinear complementarity problems. J. Optim. Theory Appl. 160(3), 711–729 (2014)
Lai, M.-J., Wang, J.: An unconstrained \(\ell _q\) minimization with \(0\le q \le 1\) for sparse solution of underdetermined linear systems. SIAM J. Optim. 21(1), 82–101 (2011)
Le Thi, H.A., Dinh, T.P., Le, H.M., Vo, X.T.: Dc approximation approaches for sparse optimization. Eur. J. Oper. Res. 244(1), 26–46 (2015)
Liuzzi, G., Rinaldi, F.: Solving \(\backslash \ell {\_}0\)-penalized problems with simple constraints via the frank-wolfe reduced dimension method. Optim. Lett. 9(1), 57–74 (2015)
Malioutov, D., Aravkin, A.: Iterative log thresholding. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7198–7202. IEEE (2014)
Mallat, S.: A Wavelet Tour of Signal Processing. Academic Press, Cambridge (1999)
Mohimani, H., Babaie-Zadeh, M., Jutten, C.: A fast approach for overcomplete sparse decomposition based on smoothed norm. IEEE Trans. Signal Process. 57(1), 289–301 (2009)
Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)
Needell, D.: Noisy signal recovery via iterative reweighted l1-minimization. In: 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers, pp. 113–117. IEEE (2009)
Needell, D., Tropp, J.A.: Cosamp: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmonic Anal. 26(3), 301–321 (2009)
Rinaldi, F.: Mathematical programming methods for minimizing the zero-norm over polyhedral sets. PhD thesis, PhD thesis, Sapienza, University of Rome (2009)
Rinaldi, F.: Concave programming for finding sparse solutions to problems with convex constraints. Optim. Methods Softw. 26(6), 971–992 (2011)
Rinaldi, F., Schoen, F., Sciandrone, M.: Concave programming for minimizing the zero-norm over polyhedral sets. Comput. Optim. Appl. 46(3), 467–486 (2010)
Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)
Tropp, J., Wright, S.J., et al.: Computational methods for sparse solution of linear inverse problems. Proc. IEEE 98(6), 948–958 (2010)
Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004)
Tropp, J.A.: Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inf. Theory 52(3), 1030–1051 (2006)
Voronin, S., Chartrand, R.: A new generalized thresholding algorithm for inverse problems with sparsity constraints. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1636–1640. IEEE (2013)
Weston, J., Elisseeff, A., Schölkopf, B., Tipping, M.: Use of the zero-norm with linear models and kernel methods. J. Mach. Learn. Res. 3(Mar), 1439–1461 (2003)
Wipf, D., Nagarajan, S.: Iterative reweighted \(\ell _1 \) and \(\ell _2 \) methods for finding sparse solutions. IEEE J. Sel. Top. Signal Process. 4(2), 317–329 (2010)
Xie, Z., Hu, J.: Rewighted l1-minimization for sparse solutions to underdetermined linear systems. In: 2013 6th International Congress on Image and Signal Processing (CISP), vol. 3, pp. 1660–1664. IEEE (2013)
Zhao, Y.-B.: Rsp-based analysis for sparsest and least \(\ell _1 \)-norm solutions to underdetermined linear systems. IEEE Trans. Signal Process. 61(22), 5777–5788 (2013)
Zhao, Y.-B., Kočvara, M.: A new computational method for the sparsest solutions to systems of linear equations. SIAM J. Optim. 25(2), 1110–1134 (2015)
Zhao, Y.-B., Luo, Z.-Q.: Constructing new weighted \(\ell _1\)-algorithms for the sparsest points of polyhedral sets. Math. Oper. Res. 42(1), 57–76 (2016)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Haddou, M., Migot, T. A smoothing method for sparse optimization over convex sets. Optim Lett 14, 1053–1069 (2020). https://doi.org/10.1007/s11590-019-01408-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11590-019-01408-x