Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

Further analysis of multilevel Stein variational gradient descent with an application to the Bayesian inference of glacier ice models

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

Multilevel Stein variational gradient descent is a method for particle-based variational inference that leverages hierarchies of surrogate target distributions with varying costs and fidelity to computationally speed up inference. The contribution of this work is twofold. First, an extension of a previous cost complexity analysis is presented that applies even when the exponential convergence rate of single-level Stein variational gradient descent depends on iteration-varying parameters. Second, multilevel Stein variational gradient descent is applied to a large-scale Bayesian inverse problem of inferring discretized basal sliding coefficient fields of the Arolla glacier ice. The numerical experiments demonstrate that the multilevel version achieves orders of magnitude speedups compared to its single-level version.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Kaipio, J., Somersalo, E.: Statistical inverse problems: Discretization, model reduction, and inverse crimes. J. Comput. Appl. Math. 198(2), 493–504 (2007)

    Article  MathSciNet  Google Scholar 

  2. Stuart, A.M.: Inverse problems: A Bayesian perspective. Acta Numer. 19, 451–559 (2010)

    Article  MathSciNet  Google Scholar 

  3. Latz, J.: On the well-posedness of Bayesian inverse problems. SIAM/ASA Journal on Uncertainty Quantification 8(1), 451–482 (2020)

    Article  MathSciNet  Google Scholar 

  4. Liu, Q., Wang, D.: Stein variational gradient descent: A general purpose Bayesian inference algorithm. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 2378–2386 (2016)

  5. Detommaso, G., Cui, T., Marzouk, Y., Spantini, A., Scheichl, R.: A Stein variational Newton method. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31, pp. 9169–9179 (2018)

  6. Duncan, A., Nüsken, N., Szpruch, L.: On the geometry of stein variational gradient descent. J. Mach. Learn. Res. 24(56), 1–39 (2023)

    MathSciNet  Google Scholar 

  7. Wang, D., Tang, Z., Bajaj, C., Liu, Q.: Stein variational gradient descent with matrix-valued kernels. In: Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32 (2019)

  8. Cui, T., Law, K.J.H., Marzouk, Y.M.: Dimension-independent likelihood-informed MCMC. J. Comput. Phys. 304, 109–137 (2016)

    Article  MathSciNet  Google Scholar 

  9. Chen, P., Wu, K., Chen, J., Leary-Roseberry, T.O., Ghattas, O.: Projected Stein variational Newton: A fast and scalable Bayesian inference method in high dimensions. In: Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32, pp. 15130–15139 (2019)

  10. Chen, P., Ghattas, O.: Stein variational reduced basis Bayesian inversion. SIAM J. Sci. Comput. 43(2), 1163–1193 (2021)

    Article  MathSciNet  Google Scholar 

  11. Liu, Q.: Stein variational gradient descent as gradient flow. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 3115–3123 (2017)

  12. Lu, J., Lu, Y., Nolen, J.: Scaling limit of the Stein variational gradient descent: The mean field regime. SIAM J. Math. Anal. 51(2), 648–671 (2019)

    Article  MathSciNet  Google Scholar 

  13. Chewi, S., Gouic, T.L., Lu, C., Maunu, T., Rigollet, P.: SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33 (2020)

  14. Korba, A., Salim, A., Arbel, M., Luise, G., Gretton, A.: A non-asymptotic analysis for Stein variational gradient descent. In: Advances in Neural Information Processing Systems, vol. 33 (2020)

  15. Ba, J., Erdogdu, M., Ghassemi, M., Suzuki, T., Wu, D.: Towards characterizing the high-dimensional bias of kernel-based particle inference algorithms. In: 2nd Symposium on Advances in Approximate Bayesian Inference, pp. 1–17 (2019)

  16. Alsup, T., Venturi, L., Peherestorfer, B.: Multilevel Stein variational gradient descent with applications to Bayesian inverse problems. In: Bruna, J., Hesthaven, J.S., Zdeborova, L. (eds.) Proceedings of Machine Learning Research. 2nd Annual Conference on Mathematical and Scientific Machine Learning, vol. 145, pp. 1–25 (2021)

  17. Peherstorfer, B., Willcox, K., Gunzburger, M.: Survey of multifidelity methods in uncertainty propagation, inference, and optimization. SIAM Rev. 60(3), 550–591 (2018)

    Article  MathSciNet  Google Scholar 

  18. Konrad, J., Farcaş, I.-G., Peherstorfer, B., Di Siena, A., Jenko, F., Neckel, T., Bungartz, H.-J.: Data-driven low-fidelity models for multi-fidelity Monte Carlo sampling in plasma micro-turbulence analysis. J. Comput. Phys. 451, 110898 (2022)

    Article  MathSciNet  Google Scholar 

  19. Heinrich, S.: Multilevel Monte Carlo methods. In: Proceedings of the Third International Conference on Large-Scale Scientific Computing-Revised Papers, LSSC ’01, pp. 58–67 (2001)

  20. Giles, M.: Multilevel Monte Carlo path simulation. Oper. Res. 56(3), 607–617 (2008)

    Article  MathSciNet  Google Scholar 

  21. Cliffe, K.A., Giles, M., Scheichl, R., Teckentrup, A.L.: Multilevel Monte Carlo methods and applications to elliptic PDEs with random coefficients. Comput. Vis. Sci. 14(1), 3–15 (2011)

    Article  MathSciNet  Google Scholar 

  22. Peherstorfer, B., Gunzburger, M., Willcox, K.: Convergence analysis of multifidelity Monte Carlo estimation. Numer. Math. 139(3), 683–707 (2018)

    Article  MathSciNet  Google Scholar 

  23. Peherstorfer, B., Beran, P.S., Willcox, K.: Multifidelity Monte Carlo estimation for large-scale uncertainty propagation. In: 2018 AIAA Non-Deterministic Approaches Conference (2018)

  24. Dodwell, T.J., Ketelsen, C., Scheichl, R., Teckentrup, A.L.: A hierarchical multilevel Markov chain Monte Carlo algorithm with applications to uncertainty quantification in subsurface flow. SIAM/ASA J. Uncertain. Quantif. 3(1), 1075–1108 (2015)

    Article  MathSciNet  Google Scholar 

  25. Lykkegaard, M.B., Dodwell, T.J., Fox, C., Mingas, G., Scheichl, R.: Multilevel delayed acceptance MCMC. SIAM/ASA J. Uncertain. Quantif. 11(1), 1–30 (2023)

    Article  MathSciNet  Google Scholar 

  26. Peherstorfer, B., Marzouk, Y.: A transport-based multifidelity preconditioner for Markov chain Monte Carlo. Adv. Comput. Math. 45, 2321–2348 (2019)

    Article  MathSciNet  Google Scholar 

  27. Beskos, A., Jasra, A., Law, K., Tempone, R., Zhou, Y.: Multilevel sequential Monte Carlo samplers. Stoch. Process. Appl. 127(5), 1417–1440 (2017)

    Article  MathSciNet  Google Scholar 

  28. Latz, J., Papaioannou, I., Ullmann, E.: Multilevel sequential\(^2\) Monte Carlo for Bayesian inverse problems. J. Comput. Phys. 368, 154–178 (2018)

    Article  MathSciNet  Google Scholar 

  29. Wagner, F., Latz, J., Papaioannou, I., Ullmann, E.: Multilevel sequential importance sampling for rare event estimation. SIAM J. Sci. Comput. 42(4), 2062–2087 (2020)

    Article  MathSciNet  Google Scholar 

  30. Peherstorfer, B., Kramer, B., Willcox, K.: Multifidelity preconditioning of the cross-entropy method for rare event simulation and failure probability estimation. SIAM/ASA J. Uncertain. Quantif. 6(2), 737–761 (2018)

    Article  MathSciNet  Google Scholar 

  31. Alsup, T., Peherstorfer, B.: Context-aware surrogate modeling for balancing approximation and sampling costs in multi-fidelity importance sampling and Bayesian inverse problems. SIAM/ASA J. Uncertain. Quantif. (2022). (accepted)

  32. Gregory, A., Cotter, C.J., Reich, S.: Multilevel ensemble transform particle filtering. SIAM J. Sci. Comput. 38(3), 1317–1338 (2016)

    Article  MathSciNet  Google Scholar 

  33. Briggs, W., Henson, V.E., McCormick, S.: A Multigrid Tutorial, Second Edition, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia (2000)

    Book  Google Scholar 

  34. Hackbush, W.: Multi-Grid Methods and Applications. Springer, Berlin (1985)

    Book  Google Scholar 

  35. Li, Z., Fan, Y., Ying, L.: Multilevel fine-tuning: Closing generalization gaps in approximation of solution maps under a limited budget for training data. Multiscale Modeling & Simulation 19(1) (2021)

  36. Hoel, H., Law, K., Tempone, R.: Multilevel ensemble Kalman filtering. SIAM J. Numer. Anal. 54(3), 1813–1839 (2016)

    Article  MathSciNet  Google Scholar 

  37. Chada, N., Jasra, A., Yu, F.: Multilevel ensemble Kalman-Bucy filters. SIAM/ASA J. Uncertain. Quantif. 10(2), 584–618 (2022)

    Article  MathSciNet  Google Scholar 

  38. Petra, N., Martin, J., Stadler, G., Ghattas, O.: A computational framework for infinite-dimensional Bayesian inverse problems, part II: Stochastic Newton MCMC with application to ice sheet flow inverse problems. SIAM J. Sci. Comput. 36(4), 1525–1555 (2014)

    Article  MathSciNet  Google Scholar 

  39. Pattyn, F., Perichon, L., Aschwanden, A., Breuer, B., Smedt, B., Gagliardini, O., Gudmundsson, G.H., Hindmarsh, R.C.A., Hubbard, A., Johnson, J.V., Kleiner, T., Konovalov, Y., Martin, C., Payne, A.J., Pollard, D., Price, S., Ruckamp, M., Saito, F., Soucek, O., Sugiyama, S., Zwinger, T.: Benchmark experiments for higher-order and full-Stokes ice sheet models (ISMIP-HOM). Cryosphere 2, 95–108 (2008)

    Article  Google Scholar 

  40. Alnaes, M.S., Blechta, J., Hake, J., Johansson, A., Kehlet, B., Logg, A., Richardson, C., Ring, J., Rognes, M.E., Wells, G.N.: The FEniCS Project Version 1.5 (2015)

  41. Villa, U., Petra, N., Ghattas, O.: Documentation to “hIPPYlib: an Extensible Software Framework for Large-scale Deterministic and Bayesian Inverse Problems”. http://hippylib.github.io (2016)

  42. Villa, U., Petra, N., Ghattas, O.: hIPPYlib: an extensible software framework for large-scale deterministic and Bayesian inverse problems. J. Open Source Softw. 3(30) (2018)

  43. Villa, U., Petra, N., Ghattas, O.: HIPPYlib: An extensible software framework for large-scale inverse problems governed by PDEs: Part I: Deterministic inversion and linearized Bayesian inference. ACM Trans. Math. Softw. 47(2) (2021)

  44. Kim, K.-T., Villa, U., Parno, M., Marzouk, Y., Ghattas, O., Petra, N.: hIPPYlib-MUQ: A Bayesian inference software framework for integration of data with complex predictive models under uncertainty. arXiv:2112.00713 (2021)

  45. Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., Smola, A.: A kernel two-sample test. J. Mach. Learn. Res. 13, 723–773 (2012)

    MathSciNet  Google Scholar 

  46. Jordan, R., Kinderlehrer, D., Otto, F.: The variational formulation of the Fokker–Planck equation. SIAM J. Math. Anal. 29(1) (1998)

  47. Cotter, S.L., Roberts, G.O., Stuart, A.M., White, D.: MCMC methods for functions: modifying old algorithms to make them faster. Statist. Sci. 28(3), 424–446 (2013)

    Article  MathSciNet  Google Scholar 

Download references

Funding

The first and third author acknowledge support from the Air Force Office of Scientific Research under Award Number FA9550-21-1-0222 (Dr. Fariba Fahroo) and the National Science Foundation (NSF) under award IIS-1901091. The second and fourth author acknowledge support provided by the NSF under Grant No. CAREER-1654311. The second author acknowledges further support provided by the NSF under Grant No. DMS-1840265.

Author information

Authors and Affiliations

  1. Courant Institute of Mathematical Sciences, New York University, 251 Mercer Street, New York, NY, 10012, USA

    Terrence Alsup & Benjamin Peherstorfer

  2. Department of Applied Mathematics, University of California, Merced, 5200 North Lake Rd, Merced, CA, 95343, USA

    Tucker Hartland & Noemi Petra

Authors
  1. Terrence Alsup
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tucker Hartland
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin Peherstorfer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Noemi Petra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benjamin Peherstorfer.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Communicated by: Anthony Nouy

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alsup, T., Hartland, T., Peherstorfer, B. et al. Further analysis of multilevel Stein variational gradient descent with an application to the Bayesian inference of glacier ice models. Adv Comput Math 50, 65 (2024). https://doi.org/10.1007/s10444-024-10153-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10444-024-10153-4

Keywords

Search

Navigation