Multilevel CNNs for parametric PDEs
Article No.: 373, Pages 17916 - 17957
Abstract
We combine concepts from multilevel solvers for partial differential equations (PDEs) with neural network based deep learning and propose a new methodology for the efficient numerical solution of high-dimensional parametric PDEs. An in-depth theoretical analysis shows that the proposed architecture is able to approximate multigrid V-cycles to arbitrary precision with the number of weights only depending logarithmically on the resolution of the finest mesh. As a consequence, approximation bounds for the solution of parametric PDEs by neural networks that are independent on the (stochastic) parameter dimension can be derived.
The performance of the proposed method is illustrated on high-dimensional parametric linear elliptic PDEs that are common benchmark problems in uncertainty quantification. We find substantial improvements over state-of-the-art deep learning-based solvers. As particularly challenging examples, random conductivity with high-dimensional nonaffine Gaussian fields in 100 parameter dimensions and a random cookie problem are examined. Due to the multilevel structure of our method, the amount of training samples can be reduced on finer levels, hence significantly lowering the generation time for training data and the training time of our method.
References
[1]
Ben Adcock and Nick Dexter. The gap between theory and practice in function approximation with deep neural networks. SIAM Journal on Mathematics of Data Science, 3(2): 624-655, 2021.
[2]
Anima Anandkumar, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
[3]
Ivo Babuska, Fabio Nobile, and Raul Tempone. A stochastic collocation method for elliptic partial differential equations with random input data. SIAM Journal on Numerical Analysis, 45:1005-1034, 2007.
[4]
Pierre Baldi. Deep learning in biomedical data science. Annual Review of Biomedical Data Science, 1(1):181-205, 2018.
[5]
Jonas Ballani, Daniel Kressner, and Michael Peters. Multilevel tensor approximation of pdes with random data. Stochastics and Partial Differential Equations: Analysis and Computations, 5:400-427, 2016.
[6]
Christian Beck, Martin Hutzenthaler, Arnulf Jentzen, and Benno Kuckuck. An overview on deep learning-based approximation methods for partial differential equations. Discrete and Continuous Dynamical Systems - B, 28(6):3697-3746, 2023. ISSN 1531-3492.
[7]
Julius Berner, Markus Dablander, and Philipp Grohs. Numerically solving parametric families of high-dimensional kolmogorov partial differential equations via deep learning. Advances in Neural Information Processing Systems, 33:16615-16627, 2020a.
[8]
Julius Berner, Philipp Grohs, and Arnulf Jentzen. Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of black-scholes partial differential equations. SIAM Journal on Mathematics of Data Science, 2(3):631-657, 2020b.
[9]
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64(2):525-545, 2019. ISSN 1432-0924.
[10]
Dietrich Braess. Finite elements: Theory, fast solvers, and applications in solid mechanics. Cambridge University Press, 2007.
[11]
Dietrich Braess and Wolfgang Hackbusch. A new convergence proof for the multigrid method including the v-cycle. Siam Journal on Numerical Analysis, 20:967-975, 1983.
[12]
Yuyan Chen, Bin Dong, and Jinchao Xu. Meta-mgnet: Meta multigrid networks for solving parameterized partial differential equations. Journal of Computational Physics, page 110996, 2022.
[13]
Abdellah Chkifa, Albert Cohen, Giovanni Migliorati, Fabio Nobile, and Raul Tempone. Discrete least squares polynomial approximation with random evaluations- application to parametric and stochastic elliptic pdes. ESAIM: Mathematical Modelling and Numerical Analysis-Modélisation Mathématique et Analyse Numérique, 49(3):815-837, 2015.
[14]
Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fine approach in single image deblurring. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4621-4630, 2021.
[15]
Philippe G. Ciarlet. The Finite Element Method for Elliptic Problems. Society for Industrial and Applied Mathematics, 2002.
[16]
Albert Cohen and Ronald DeVore. Approximation of high-dimensional parametric pdes. Acta Numerica, 24:1-159, 2015.
[17]
Albert Cohen and Giovanni Migliorati. Near-optimal approximation methods for elliptic pdes with lognormal coefficients. Mathematics of Computation, 2023.
[18]
Weinan E, Jiequn Han, and Arnulf Jentzen. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics, 5(4):349-380, 2017. ISSN 2194-671X.
[19]
Martin Eigel, Claude Jeffrey Gittelson, Christoph Schwab, and Elmar Zander. Adaptive stochastic galerkin FEM. Computer Methods in Applied Mechanics and Engineering, 270: 247-269, March 2014.
[20]
Martin Eigel, Reinhold Schneider, Philipp Trunschke, and Sebastian Wolf. Variational monte carlo--bridging concepts of machine learning and high-dimensional partial differential equations. Advances in Computational Mathematics, 45(5-6):2503-2532, October 2019a.
[21]
Martin Eigel, Reinhold Schneider, Philipp Trunschke, and Sebastian Wolf. Variational monte carlo--bridging concepts of machine learning and high-dimensional partial differential equations. Advances in Computational Mathematics, 45(5):2503-2532, 2019b.
[22]
Martin Eigel, Manuel Marschall, Max Pfeffer, and Reinhold Schneider. Adaptive stochastic galerkin FEM for lognormal coefficients in hierarchical tensor representations. Numerische Mathematik, 145(3):655-692, 6 2020.
[23]
Oliver G Ernst, Bjorn Sprungk, and Lorenzo Tamellini. Convergence of sparse collocation for functions of countably many gaussian random variables (with application to elliptic pdes). SIAM Journal on Numerical Analysis, 56(2):877-905, 2018.
[24]
Yuwei Fan, Jordi Feliu-Faba, Lin Lin, Lexing Ying, and Leonardo Zepeda-Núnez. A multiscale neural network based on hierarchical nested bases. Research in the Mathematical Sciences, 6(2):1-28, 2019a.
[25]
Yuwei Fan, Lin Lin, Lexing Ying, and Leonardo Zepeda-Núnez. A multiscale neural network based on hierarchical matrices. Multiscale Modeling & Simulation, 17(4):1189-1213, 2019b.
[26]
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative convnets via multi-grid modeling and sampling. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9155-9164, 2018.
[27]
Moritz Geist, Philipp Petersen, Mones Raslan, Reinhold Schneider, and Gitta Kutyniok. Numerical solution of the parametric diffusion equation by deep neural networks. Journal of Scientific Computing, 88(1):1-37, 2021.
[28]
Martin Genzel, Ingo Gühring, Jan MacDonald, and Maximilian März. Near-exact recovery for tomographic inverse problems via deep learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 7368-7381. PMLR, 2022.
[29]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT Press, Cambridge, 2016.
[30]
Philipp Grohs and Lukas Herrmann. Deep neural network approximation for high-dimensional elliptic pdes with boundary conditions. IMA Journal of Numerical Analysis, 42(3):2055-2082, 2022.
[31]
Philipp Grohs, Fabian Hornung, Arnulf Jentzen, and Philippe Von Wurstemberger. A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black-scholes partial differential equations. arXiv preprint arXiv:1809.02362, 2018.
[32]
Tamara G Grossmann, Urszula Julia Komorowska, Jonas Latz, and Carola-Bibiane Schönlieb. Can physics-informed neural networks beat the finite element method? arXiv preprint arXiv:2302.04107, 2023.
[33]
Ingo Gühring and Mones Raslan. Approximation rates for neural networks with encodable weights in smoothness spaces. Neural Networks, 134:107-130, 2021.
[34]
Xiaoxiao Guo, Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow approximation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 481-490, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322.
[35]
Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep ReLU neural networks in Ws,p norms. Analysis and Applications (Singap.), 18(05): 803-859, 2020.
[36]
Ingo Gühring, Mones Raslan, and Gitta Kutyniok. Expressivity of deep neural networks. In Philipp Grohs and Gitta Kutyniok, editors, Mathematical Aspects of Deep Learning, page 149-199. Cambridge University Press, 2022.
[37]
Wolfgang Hackbusch. Multi-grid methods and applications, volume 4. Springer Science & Business Media, 2013.
[38]
Jiequn Han, Arnulf Jentzen, et al. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in mathematics and statistics, 5(4):349-380, 2017.
[39]
Helmut Harbrecht, Michael Peters, and Markus Siebenmorgen. Multilevel accelerated quadrature for pdes with log-normally distributed diffusion coefficient. SIAM/ASA Journal on Uncertainty Quantification, 4(1):520-551, 2016.
[40]
Juncai He and Jinchao Xu. Mgnet: A unified framework of multigrid and convolutional neural network. Science China Mathematics, 62(7):1331-1354, 2019. ISSN 1869-1862.
[41]
J.S. Hesthaven and S. Ubbiali. Non-intrusive reduced order modeling of nonlinear problems using neural networks. Journal of Computational Physics, 363:55-78, 2018. ISSN 0021- 9991.
[42]
Zhihao Jiang, Pejman Tahmasebi, and Zhiqiang Mao. Deep residual u-net convolution neural networks with autoregressive strategy for fluid flow predictions in large-scale geosystems. Advances in Water Resources, 150:103878, 02 2021.
[43]
John A Keith, Valentin Vassilev-Galindo, Bingqing Cheng, Stefan Chmiela, Michael Gastegger, Klaus-Robert Müller, and Alexandre Tkatchenko. Combining machine learning and computational chemistry for predictive insights into chemical systems. Chemical Reviews, 121(16):9816-9872, 2021.
[44]
Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. Variational physicsinformed neural networks for solving partial differential equations. arXiv preprint arXiv:1912.00873, 2019.
[45]
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[46]
Omar M Knio and OP Le Maitre. Uncertainty propagation in cfd using polynomial chaos decomposition. Fluid dynamics research, 38(9):616, 2006.
[47]
Gitta Kutyniok, Philipp Petersen, Mones Raslan, and Reinhold Schneider. A theoretical analysis of deep neural networks and parametric pdes. Constructive Approximation, 55 (1):73-125, 2022.
[48]
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755-6766, 2020.
[49]
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021a.
[50]
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021b.
[51]
Liangliang Liu, Jianhong Cheng, Quan Quan, Fang-Xiang Wu, Yu-Ping Wang, and Jianxin Wang. A survey on u-shaped networks in medical image segmentations. Neurocomputing, 409:244-258, 2020. ISSN 0925-2312.
[52]
Anders Logg and Garth N. Wells. Dolfin. ACM Transactions on Mathematical Software, 37(2):1-28, 2010. ISSN 1557-7295.
[53]
Gabriel J Lord, Catherine E Powell, and Tony Shardlow. An introduction to computational stochastic PDEs, volume 50. Cambridge University Press, 2014.
[54]
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM review, 63(1):208-228, 2021.
[55]
Kjetil O Lye, Siddhartha Mishra, and Roberto Molinaro. A multi-level procedure for enhancing accuracy of machine learning algorithms. European Journal of Applied Mathematics, 32(3):436-469, 2021.
[56]
Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physics-informed neural networks for approximating PDEs. IMA Journal of Numerical Analysis, 43(1):1-43, 01 2022. ISSN 0272-4979.
[57]
Fabio Nobile, Raúl Tempone, and Clayton G Webster. A sparse grid stochastic collocation method for partial differential equations with random input data. SIAM Journal on Numerical Analysis, 46(5):2309-2345, 2008.
[58]
Frank Noé, Alexandre Tkatchenko, Klaus-Robert Müller, and Cecilia Clementi. Machine learning for molecular simulation. Annual Review of Physical Chemistry, 71(1):361-390, 2020.
[59]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[60]
Philipp Petersen and Felix Voigtländer. Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Networks, 108:296-330, 2018.
[61]
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-9991.
[62]
Amuthan A Ramabathiran and Prabhu Ramachandran. Spinn: sparse, physics-based, and partially interpretable neural networks for pdes. Journal of Computational Physics, 445: 110600, 2021.
[63]
Lewis Fry Richardson and Richard Tetley Glazebrook. Ix. the approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application to the stresses in a masonry dam. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 210(459-470):307-357, 1911.
[64]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[65]
Youcef Saad and Martin H Schultz. Gmres: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific Computing, 7(3): 856-869, 1986. ISSN 0196-5204.
[66]
Peter Sadowski and Pierre Baldi. Deep learning in the natural sciences: Applications to physics. In Lev Rozonoer, Boris Mirkin, and Ilya Muchnik, editors, Braverman Readings in Machine Learning. Key Ideas from Inception to Current State: International Conference Commemorating the 40th Anniversary of Emmanuil Braverman's Decease, Boston, MA, USA, April 28-30, 2017, Invited Talks, pages 269-297. Springer International Publishing, Cham, 2018.
[67]
Christoph Schwab and Claude Jeffrey Gittelson. Sparse tensor discretizations of highdimensional parametric and stochastic pdes. Acta Numerica, 20:291-467, 2011.
[68]
Christoph Schwab and Jakob Zech. Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in uq. Analysis and Applications, 17(01):19-55, 2019.
[69]
W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z.Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874-1883, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.
[70]
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364, 2018. ISSN 0021-9991.
[71]
A. L. Teckentrup, P. Jantsch, C. G. Webster, and M. Gunzburger. A multilevel stochastic collocation method for partial differential equations with random input data. SIAM/ASA Journal on Uncertainty Quantification, 3(1):1046-1074, 2015.
[72]
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Representations, 2020.
[73]
Xuefeng Xu and Chen-Song Zhang. Convergence analysis of inexact two-grid methods: A theoretical framework. SIAM Journal on Numerical Analysis, 60(1):133-156, 2022.
[74]
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[75]
Harry Yserentant. Old and new convergence proofs for multigrid methods. Acta numerica, 2:285-326, 1993.
[76]
Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
[77]
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415-447, 2018. ISSN 0021-9991.
Index Terms
- Multilevel CNNs for parametric PDEs
Index terms have been assigned to the content through auto-classification.
Recommendations
Multi-index Stochastic Collocation Convergence Rates for Random PDEs with Parametric Regularity
We analyze the recent Multi-index Stochastic Collocation (MISC) method for computing statistics of the solution of a partial differential equation (PDE) with random data, where the random coefficient is parametrized by means of a countable sequence of ...
Discontinuous Galerkin Methods for Friedrichs’ Systems. Part II. Second-order Elliptic PDEs
This paper is the second part of a work attempting to give a unified analysis of discontinuous Galerkin methods. The setting under scrutiny is that of Friedrichs’ systems endowed with a particular $2 \times 2$ structure in which one unknown can be ...
Solving PDEs with the aid of two-dimensional Haar wavelets
Two-dimensional Haar wavelets are applied for solution of the partial differential equations (PDEs). The proposed method is mathematically simple and fast. To demonstrate the efficiency of the method, two test problems (solution of the diffusion and ...
Comments
Information & Contributors
Information
Published In
January 2023
18881 pages
Copyright © 2023.
CC-BY 4.0
Publisher
JMLR.org
Publication History
Published: 06 March 2024
Accepted: 01 December 2023
Revised: 01 November 2023
Received: 01 April 2023
Published in JMLR Volume 24, Issue 1
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 16Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)2
Reflects downloads up to 03 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in