research-article

Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry property

Authors:

Rohit KhandekarAuthors Info & Claims

ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning

Pages 337 - 344

https://doi.org/10.1145/1553374.1553417

Published: 14 June 2009 Publication History

Abstract

We present an algorithm for finding an s-sparse vector x that minimizes the square-error ∥y -- Φx∥² where Φ satisfies the restricted isometry property (RIP), with isometric constant δ_2s < 1/3. Our algorithm, called GraDeS (Gradient Descent with Sparsification) iteratively updates x as: [EQUATION]

where γ > 1 and H_s sets all but s largest magnitude coordinates to zero. GraDeS converges to the correct solution in constant number of iterations. The condition δ_2s < 1/3 is most general for which a near-linear time algorithm is known. In comparison, the best condition under which a polynomial-time algorithm is known, is δ_2s < √2 -- 1.

Our Matlab implementation of GraDeS outperforms previously proposed algorithms like Subspace Pursuit, StOMP, OMP, and Lasso by an order of magnitude. Curiously, our experiments also uncovered cases where L1-regularized regression (Lasso) fails but GraDeS finds the correct solution.

References

[1]

Berinde, R., Indyk, P., & Ruzic, M. (2008). Practical near-optimal sparse recovery in the L1 norm. Allerton Conference on Communication, Control, and Computing. Monticello, IL.

[2]

Blumensath, T., & Davies, M. E. (2008). Iterative hard thresholding for compressed sensing. Preprint.

[3]

Candès, E., & Wakin, M. (2008). An introduction to compressive sampling. IEEE Signal Processing Magazine, 25, 21--30.

[4]

Candès, E. J. (2008). The restricted isometry property and its implications for compressed sensing. Compte Rendus de l'Academie des Sciences, Paris, 1, 589--592.

[5]

Candès, E. J., & Romberg, J. (2004). Practical signal recovery from random projections. SPIN Conference on Wavelet Applications in Signal and Image Processing.

[6]

Candès, E. J., Romberg, J. K., & Tao, T. (2006). Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52, 489--509.

Digital Library

[7]

Candès, E. J., & Tao, T. (2005). Decoding by linear programming. IEEE Transactions on Information Theory, 51, 4203--4215.

Digital Library

[8]

Candès, E. J., & Tao, T. (2006). Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE Transactions on Information Theory, 52, 5406--5425.

Digital Library

[9]

Chen, S. S., Donoho, D. L., & Saunders, M. A. (2001). Atomic decomposition by basis pursuit. SIAM Review, 43, 129--159.

Digital Library

[10]

Dai, W., & Milenkovic, O. (2008). Subspace Pursuit for Compressive Sensing: Closing the Gap Between Performance and Complexity. ArXiv e-prints.

[11]

Do, T. T., Gan, L., Nguyen, N., & Tran, T. D. (2008). Sparsity adaptive matching pursuit algorithm for practical compressed sensing. Asilomar Conference on Signals, Systems, and Computers. Pacific Grove, California.

[12]

Donoho, D., & Others (2009). SparseLab: Seeking sparse solutions to linear systems of equations. http://sparselab.stanford.edu/.

[13]

Donoho, D. L., & Tsaig, Y. (2006). Fast solution of L1 minimization problems when the solution may be sparse. Technical Report, Institute for Computational and Mathematical Engineering, Stanford University.

[14]

Donoho, D. L., Tsaig, Y., Drori, I., & Starck, J.-L. (2006). Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit. Preprint.

[15]

Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32, 407--499.

[16]

Figueiredo, M. A. T., & Nowak, R. D. (2005). A bound optimization approach to wavelet-based image deconvolution. IEEE International Conference on Image Processing (ICIP) (pp. 782--785).

[17]

Golub, G., & Loan, C. V. (1996). Matrix computations, 3rd ed. Johns Hopkins University Press.

Digital Library

[18]

Jung, H., Ye, J. C., & Kim, E. Y. (2007). Improved k-t BLASK and k-t SENSE using FOCUSS. Phys. Med. Biol., 52, 3201--3226.

[19]

Lustig, M. (2008). Sparse MRI. Ph.D Thesis, Stanford University.

[20]

Ma, S., Yin, W., Zhang, Y., & Chakraborty, A. (2008). An efficient algorithm for compressed MR imaging using total variation and wavelets. IEEE Confererence on Computer Vision and Pattern Recognition (CVPR) (pp. 1--8).

[21]

Mallat, S. G., & Zhang, Z. (1993). Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing, 3397--3415.

Digital Library

[22]

Natarajan, B. K. (1995). Sparse approximate solutions to linear systems. SIAM Journal of Computing, 24, 227--234.

Digital Library

[23]

Needell, & Tropp, J. A. (2008). CoSaMP: Iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis, 26, 301--321.

[24]

Needell, D., & Vershynin, R. (2009). Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Foundations of Computational Mathematics, 9, 317--334.

Digital Library

[25]

Neylon, T. (2006). Sparse solutions for linear prediction problems. Doctoral dissertation, Courant Institute, New York University.

Digital Library

[26]

Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2007). Sparse feature learning for deep belief networks. In Advances in neural information processing systems 20 (NIPS), 1185--1192. MIT Press.

[27]

Sarvotham, S., Baron, D., & Baraniuk, R. (2006). Sudocodes - fast measurement and reconstruction of sparse signals. IEEE International Symposium on Information Theory (ISIT) (pp. 2804--2808). Seattle, Washington.

[28]

Tropp, J. A., & Gilbert, A. C. (2007). Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Info. Theory, 53, 4655--4666.

Digital Library

[29]

Wainwright, M. J., Ravikumar, P., & Lafferty, J. D. (2006). High-dimensional graphical model selection using l ₁-regularized logistic regression. In Advances in neural information processing systems 19 (NIPS'06), 1465--1472. Cambridge, MA: MIT Press.

[30]

Zou, H., Hastie, T., & Tibshirani, R. (2006). Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15, 262--286.

Cited By

Kang YChoi W(2024)MCMC Sampling-Based Randomized Likelihood Decoding for Sparse RecoveryIEEE Transactions on Communications10.1109/TCOMM.2023.332465872:1(465-479)Online publication date: Jan-2024
https://doi.org/10.1109/TCOMM.2023.3324658
Li CMa ZSun DZhang GWen J(2024)Stochastic IHT With Stochastic Polyak Step-Size for Sparse Signal RecoveryIEEE Signal Processing Letters10.1109/LSP.2024.342635331(2035-2039)Online publication date: 2024
https://doi.org/10.1109/LSP.2024.3426353
He ZShu QWen JSo H(2024)A Novel Iterative Thresholding Algorithm for Arctangent Regularization ProblemICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10448179(9651-9655)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10448179
Show More Cited By

Index Terms

Recommendations

Gradient descent with proximal average for nonconvex and composite regularization
AAAI'14: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence

Sparse modeling has been highly successful in many realworld applications. While a lot of interests have been on convex regularization, recent studies show that nonconvex regularizers can outperform their convex counterparts in many situations. However, ...
Descent gradient methods for nonsmooth minimization problems in ill-posed problems

Descent gradient methods are the most frequently used algorithms for computing regularizers of inverse problems. They are either directly applied to the discrepancy term, which measures the difference between operator evaluation and data or to a ...
A coordinate gradient descent method for l1-regularized convex minimization

In applications such as signal processing and statistics, many problems involve finding sparse solutions to under-determined linear systems of equations. These problems can be formulated as a structured nonsmooth optimization problems, i.e., the problem ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning

June 2009

1331 pages

ISBN:9781605585161

DOI:10.1145/1553374

General Chair:
Andrea Danyluk
Williams College
,
Program Chairs:
Léon Bottou
NEC Laboratories America
,
Michael Littman
Rutgers University

Copyright © 2009 Copyright 2009 by the author(s)/owner(s).

Sponsors

NSF
Microsoft Research: Microsoft Research
MITACS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 June 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ICML '09

Sponsor:

Microsoft Research

ICML '09: The 26th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

June 14 - 18, 2009

Quebec, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

90
Total Citations
View Citations
901
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)3

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kang YChoi W(2024)MCMC Sampling-Based Randomized Likelihood Decoding for Sparse RecoveryIEEE Transactions on Communications10.1109/TCOMM.2023.332465872:1(465-479)Online publication date: Jan-2024
https://doi.org/10.1109/TCOMM.2023.3324658
Li CMa ZSun DZhang GWen J(2024)Stochastic IHT With Stochastic Polyak Step-Size for Sparse Signal RecoveryIEEE Signal Processing Letters10.1109/LSP.2024.342635331(2035-2039)Online publication date: 2024
https://doi.org/10.1109/LSP.2024.3426353
He ZShu QWen JSo H(2024)A Novel Iterative Thresholding Algorithm for Arctangent Regularization ProblemICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10448179(9651-9655)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10448179
Xie LWang J(2024)Sufficient condition based on nearly optimal order RIC for IHT algorithmApplicable Analysis10.1080/00036811.2024.2311697(1-10)Online publication date: 31-Jan-2024
https://doi.org/10.1080/00036811.2024.2311697
Wen JHe HHe ZZhu F(2023)A Pseudo-Inverse-Based Hard Thresholding Algorithm for Sparse Signal RecoveryIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2022.317286824:7(7621-7630)Online publication date: Jul-2023
https://doi.org/10.1109/TITS.2022.3172868
He ZShu QWang YWen J(2023)A ReLU-based hard-thresholding algorithm for non-negative sparse signal recoverySignal Processing10.1016/j.sigpro.2023.109260(109260)Online publication date: Sep-2023
https://doi.org/10.1016/j.sigpro.2023.109260
Sun ZZhou JZhao YMeng N(2023)Heavy-ball-based hard thresholding algorithms for sparse signal recoveryJournal of Computational and Applied Mathematics10.1016/j.cam.2023.115264430(115264)Online publication date: Oct-2023
https://doi.org/10.1016/j.cam.2023.115264
Muzaffar MSharqi R(2023)A review of spectrum sensing in modern cognitive radio networksTelecommunication Systems10.1007/s11235-023-01079-185:2(347-363)Online publication date: 24-Nov-2023
https://doi.org/10.1007/s11235-023-01079-1
Yuan XLi P(2022)Stability and Risk Bounds of Iterative Hard ThresholdingIEEE Transactions on Information Theory10.1109/TIT.2022.317043468:10(6663-6681)Online publication date: Oct-2022
https://doi.org/10.1109/TIT.2022.3170434
Han THao KTang XCai XWang TLiu X(2022)A Compressed Sensing Network for Acquiring Human Pressure InformationIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2020.304142214:2(388-402)Online publication date: Jun-2022
https://doi.org/10.1109/TCDS.2020.3041422
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents