Abstract
Numerical simulation processes in scientific and engineering applications require efficient solutions of large sparse linear systems, and variants of Krylov subspace solvers with various preconditioning techniques have been developed. However, it is time-consuming for practitioners with trial and error to find a high-performance Krylov solver in a candidate solver set for a given linear system. Therefore, it is instructive to select an efficient solver intelligently among a solver set rather than exploratory application of all solvers to solve the linear system. One promising direction of solver selection is to apply machine learning methods to construct a mapping from the matrix features to the candidate solvers. However, the computation of some matrix features is quite difficult. In this paper, we design a new selection strategy of matrix features to reduce computing cost, and then employ the selected features to construct a machine learning classifier to predict an appropriate solver for a given linear system. Numerical experiments on two attractive GMRES-type solvers for solving linear systems from the University of Florida Sparse Matrix Collection and Matrix Market verify the efficiency of our strategy, not only reducing the computing time for obtaining features and construction time of classifier but also keeping more than 90% prediction accuracy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Code Availability
The codes associated with our paper could be accessed via https://codeocean.com/capsule/2650212/tree.
References
Agullo, E., Giraud, L., & Jing, Y.-F. (2014). Block GMRES method with inexact breakdowns and deflated restarting. SIAM Journal on Matrix Analysis and Application, 35(4), 1625–1651.
Baker, A. H., Jessup, E. R., & Manteuffel, T. (2005). A technique for accelerating the convergence of restarted GMRES. SIAM Journal on Matrix Analysis and Applications, 26, 962–984.
Benzi, M. (2002). Preconditioning techniques for large linear systems: A survey. Journal of Computational Physics, 182, 418–477.
Bhowmick, S., Eijkhout, V., Freund, Y., Fuentes, E., & Keyes, D. (2000). Application of machine learning in selecting sparse linear solvers. Astronomical Journal, 119(2), 936–944.
Bhowmick, S., Eijkhout, V., Freund, Y., Fuentes, E., & Keyes, D. (2011). Application of alternating decision trees in selecting sparse linear solvers. New York: Springer.
Chen, G., & Deng, Y. (2011). Some new methods of fitness function construction in feature selection of genetic algorithm and applications. Mechanical Science and Technology, 30(1), 124–128+132.
Chen, T.-Q., & Carlos, G. (2016). XGBoost: A scalable tree boosting system. Association for Computing Machinery.
Cipra, B.A. (2000). The best of the 20th century: Editors name top 10 algorithms. SIAM News.,33(4).
Darnella, D., Morgan, R. B., & Wilcox, W. (2008). Deflated GMRES for systems with multiple shifts and multiple right-hand sides. Linear Algebra and Its Applications, 429, 2415–2434.
Davis, T. A., & Hu, Y. (2011). The University of Florida Sparse Matrix Collection. ACM Transactions on Mathematical Software, 38, 1–25.
Eijkhout, V., & Fuentes, E. (2003). A standard and software for numerical metadata. Acm Transactions on Mathematical Software, 35(4), 1–20.
Freund, Y., & Mason, L. (1999). The alternating decision tree learning algorithm. In ICML’99: Proceedings of the Sixteenth International Conference on Machine Learning, pp. 124–133.
Gaul, A., Gutknecht, M. H., Liesen, J., & Nabben, R. (2013). A framework for deflated and augmented Krylov subpace methods. SIAM Journal on Matrix Analysis and Applications, 34(2), 495–518.
Géron, A. (2019). Hands-on machine learning with Scikit-learn, Keras, and Tensorflow: Concepts, tools, and techniques to build intelligent systems (Second ed.). O’Reilly Media, Inc.
Giraud, L., Jing, Y.-F., & Xiang, Y.-F. (2022). A block minimum residual norm subspace solver with partial convergence management for sequences of linear systems. SIAM Journal on Matrix Analysis and Applications, 43(2), 710–739.
Gutknecht, M. H. (2014). Deflated and augmented Krylov subspace methods: A framework for deflated BiCG and related solvers. SIAM Journal on Matrix Analysis and Applications, 35, 1444–1466.
Holloway, A., & Chen, T.Y. (2007). Neural networks for predicting the behavior of preconditioned iterative solvers.
Jessup, E., Motter, P., Norris, B., & Sood, K. (2016). Performance-based numerical solver selection in the Lighthouse framework. SIAM Journal of Scientific Computing, 38(5), S750–S771.
John, G. H., Kohavi, R., & Pfleger, K. (1994). Irrelevant features and the subset selection problem. In Machine Learning Proceedings. San Francisco (CA): Morgan Kaufmann.
Kohavi, R., & John, G. H. (1997). Wrappers for feature subset selection. Artificial Intelligence, 97, 273–324.
Lee, H. (2012). Statistical learning method (2nd ed.). PeKing: Tsinghua University.
Liu, H., & Yu, L. (2005). Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering, 17(4), 491–502.
Morgan, R. B. (2002). GMRES with deflated restarting. SIAM Journal on Scientific Computing, 24(1), 20–37.
Morgan, R. B. (2005). Restarted block GMRES with deflation of eigenvalues. Applied Numerical Mathematics, 54(2), 222–236.
Motter, P., Sood, K., & Jessup, E. (2015). Lighthouse: An automated solver selection tool. Software Engineering for High Performance Computing in Computational Science and Engineering.
National Institute of Standards and Technology (2021). Matrix Market. available online at https://math.nist.gov/MatrixMarket/.
Saad, Y. (2003). Iterative methods for sparse linear systems (2nd ed.). Philadelphia: SIAM.
Saad, Y. (2023). The origin and development of Krylov subspace methods. Computing in Science and Engineering, 24(4), 28–39.
Saad, Y., & Schultz, M. H. (1986). GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal of Scientific and Statistical Computing, 123, 1–33.
Saad, Y., & van der Vorst, H. A. (2000). Iterative solution of linear systems in the 20th century. Journal of Computational and Applied Mathematics, 123(1–2), 1–33.
Si, S.-K. & Sun, X.-J. (2021). Mathematical modeling algorithms and applications (Third ed.). National Defence Industry Press.
Simoncini, V., & Szyld, D. B. (2007). Numerical linear algebra with applications, 14(1), 1–59.
Sood, K. (2019). Iterative solver selection techniques for sparse linear systems. Ph. D. thesis, University of Oregon.
Sood, K., Norris, B. & Jessup, E. (2015). Lighthouse: A taxonomy-based solver selection tool. In Proceedings of the 2nd International Workshop on Software Engineering for Parallel Systems, pp. 60–70.
Sood, K., Norris, B., & Jessup, E. (2017). Comparative performance modeling of parallel preconditioned Krylov methods. 2017 IEEE 19th International Conference on High Performance Computing and Communications., 26–33.
Toth, B.A. (2009). Cost effective machine learning approaches for linear solver selection. Master’s thesis, The Pennsylvania State University.
Witten, I.H., Frank, E. & Hall, M.A. (2011). Data mining: Practical machine learning tools and techniques (Third ed.). Elsevier.
Xia, G.-M., & Zeng, J.-C. (2007). A stochastic particle swarm optimization algorithm based on roulette selection genetic algorithm. Computer Engineering and Science, 150(06), 51–54.
Xu, S. (2005). Study and design of an intelligent preconditioner recommendation system. Ph. D. thesis, University of Kentucky, Lexington, Kentucky.
Xu, S. & Zhang, J. (2005). A data mining approach to matrix preconditioning problem. Technical report, University of Kentucky.
Zhang, J. (2001). Performance of ILU preconditioners for stationary 3D Navier-Stokes simulation and the matrix mining project. Proceedings of the 2001 International Conference on Preconditioning Techniques for Large Sparse Matrix Problems in Scientific and Industrial Applications.
Zou, H.-F. (2023a). The study of intelligent iterative methods for sparse linear algebraic equations based on graph neural networks. Ph. D. thesis, China Academy of Engineering Physis.
Zou, Q.-M. (2023). GMRES algorithms over 35 years. Applied Mathematics and Computation, 445(3), 127869.
Zou, H.-F., Xu, X.-W., Zhang, C.-S. & Mo, Z.-Y. (2023). AutoAMG(\(\theta \)): An auto-tuned AMG method based on deep learning for strong threshold. ArXiv, abs/2307.09879.
Acknowledgements
The authors would like to thank Editor-in-Chief Professor Paul McNicholas, the Associate Editor, and the anonymous reviewers for their valuable suggestions to improve the quality of our paper.
Funding
The research is supported by the National Natural Science Foundation of China (12071062), Science Challenge Project (TZ2016002), Science Fund for Distinguished Young Scholars of Sichuan Province (2023NSFSC1920).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A
Appendix A
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Sun, HB., Jing, YF. & Xu, XW. A New Matrix Feature Selection Strategy in Machine Learning Models for Certain Krylov Solver Prediction. J Classif (2024). https://doi.org/10.1007/s00357-024-09484-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s00357-024-09484-0