Conjugate Gradients Acceleration of Coordinate Descent for Linear Systems
Author: 
Dan GordonAuthors Info & Claims
Dan GordonAuthors Info & ClaimsAbstract
This paper introduces a conjugate gradients (CG) acceleration of the coordinate descent algorithm (CD) for linear systems. It is shown that the Kaczmarz algorithm (KACZ) can simulate CD exactly, so CD can be accelerated by CG similarly to the CG acceleration of KACZ (Björck and Elfving in BIT 19:145–163, 1979). Experimental results were carried out on large sets of problems of reconstructing bandlimited functions from random sampling. The randomness causes extreme variance between different instances of these problems, thus causing extreme variance in the advantage of CGCD over CD. The reduction of the number of iterations by CGCD varies from about 50–90% and beyond. The implementation of CGCD is simple. CGCD can also be used for the parallel solution of linear systems derived from partial differential equations, and for the efficient solution of multiple right-hand-side problems and matrix inversion.
References
[1]
Bertrand, Q., Massias, M.: Anderson acceleration of coordinate descent. In: Proceedings of International Conference on Artificial Intelligence and Statistics, vol. 130, no. 10 (2021)
[2]
Björck Å and Elfving T Accelerated projection methods for computing pseudoinverse solutions of systems of linear equations BIT 1979 19 145-163
[3]
Feichtinger H and Gröchenig K Frazier M Theory and practice of irregular sampling Wavelets: Mathematics and Applications 1994 Boca Raton CRC Press 305-363
[4]
Gordon D A derandomization approach to recovering bandlimited signals across a wide range of random sampling rates Numer. Algorithms 2018 77 4 1141-1157
[5]
Gordon D and Gordon R CGMN revisited: robust and efficient solution of stiff linear systems derived from elliptic partial differential equations ACM Trans. Math. Softw. 2008 35 3 18:1-18:27
[6]
Gordon D and Gordon R CARP-CG: a robust and efficient parallel solver for linear systems, applied to strongly convection-dominated PDEs Parallel Comput. 2010 36 9 495-515
[7]
Kaczmarz S Angenäherte Auflösung von Systemen linearer Gleichungen Bull. l’Acad. Pol. Sci. Lett. 1937 A35 355-357
[8]
Lee, Y.T., Sidford, A.: Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems. In: Proceedings of 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, FOCS 2013, pp. 147—156. IEEE Computer Society, USA (2013)
[9]
Nesterov YE Efficiency of coordinate descent methods on huge-scale optimization problems SIAM J. Optim. 2012 22 2 341-362
[10]
Nesterov YE and Stich SU Efficiency of the accelerated coordinate descent method on structured optimization problems SIAM J. Optim. 2017 27 1 110-123
[11]
Ruhe A Numerical aspects of Gram–Schmidt orthogonalization of vectors Linear Algebra Appl. 1983 52 53 591-601
[12]
Saad Y Iterative Methods for Sparse Linear Systems 2003 2 Philadelphia SIAM
[13]
Shi, H.-J.M., Tu, S., Xu, Y., Yin, W.: A primer on coordinate descent algorithms (2016)
[14]
Strohmer T and Vershynin R A randomized Kaczmarz algorithm with exponential convergence J. Fourier Anal. Appl. 2009 15 262-278
[15]
Wang Q, Li W, Bao W, and Zhang F Accelerated randomized coordinate descent for solving linear systems Mathematics 2022 10 22 4379
[16]
Wright SJ Coordinate descent algorithms Math. Program. 2015 151 1 3-34
Index Terms
- Conjugate Gradients Acceleration of Coordinate Descent for Linear Systems
Index terms have been assigned to the content through auto-classification.
Recommendations
Flexible Conjugate Gradients
We analyze the conjugate gradient (CG) method with preconditioning slightly variable from one iteration to the next. To maintain the optimal convergence properties, we consider a variant proposed by Axelsson that performs an explicit orthogonalization of ...
A coordinate gradient descent method for nonsmooth separable minimization
We consider the problem of minimizing the sum of a smooth function and a separable convex function. This problem includes as special cases bound-constrained optimization and smooth optimization with ℓ1-regularization. We propose a (block) coordinate ...
Steepest Descent and Conjugate Gradient Methods with Variable Preconditioning
We analyze the conjugate gradient (CG) method with variable preconditioning for solving a linear system with a real symmetric positive definite (SPD) matrix of coefficients $A$. We assume that the preconditioner is SPD on each step, and that the ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Plenum Press
United States
Publication History
Published: 05 August 2023
Accepted: 21 July 2023
Revision received: 24 May 2023
Received: 13 December 2022
Author Tags
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in