-
Fourier Neural Operators for Learning Dynamics in Quantum Spin Systems
Authors:
Freya Shah,
Taylor L. Patti,
Julius Berner,
Bahareh Tolooshams,
Jean Kossaifi,
Anima Anandkumar
Abstract:
Fourier Neural Operators (FNOs) excel on tasks using functional data, such as those originating from partial differential equations. Such characteristics render them an effective approach for simulating the time evolution of quantum wavefunctions, which is a computationally challenging, yet coveted task for understanding quantum systems. In this manuscript, we use FNOs to model the evolution of ra…
▽ More
Fourier Neural Operators (FNOs) excel on tasks using functional data, such as those originating from partial differential equations. Such characteristics render them an effective approach for simulating the time evolution of quantum wavefunctions, which is a computationally challenging, yet coveted task for understanding quantum systems. In this manuscript, we use FNOs to model the evolution of random quantum spin systems, so chosen due to their representative quantum dynamics and minimal symmetry. We explore two distinct FNO architectures and examine their performance for learning and predicting time evolution using both random and low-energy input states. Additionally, we apply FNOs to a compact set of Hamiltonian observables ($\sim\text{poly}(n)$) instead of the entire $2^n$ quantum wavefunction, which greatly reduces the size of our inputs and outputs and, consequently, the requisite dimensions of the resulting FNOs. Moreover, this Hamiltonian observable-based method demonstrates that FNOs can effectively distill information from high-dimensional spaces into lower-dimensional spaces. The extrapolation of Hamiltonian observables to times later than those used in training is of particular interest, as this stands to fundamentally increase the simulatability of quantum systems past both the coherence times of contemporary quantum architectures and the circuit-depths of tractable tensor networks.
△ Less
Submitted 5 September, 2024;
originally announced September 2024.
-
Beyond Closure Models: Learning Chaotic-Systems via Physics-Informed Neural Operators
Authors:
Chuwei Wang,
Julius Berner,
Zongyi Li,
Di Zhou,
Jiayun Wang,
Jane Bae,
Anima Anandkumar
Abstract:
Accurately predicting the long-term behavior of chaotic systems is crucial for various applications such as climate modeling. However, achieving such predictions typically requires iterative computations over a dense spatiotemporal grid to account for the unstable nature of chaotic systems, which is expensive and impractical in many real-world situations. An alternative approach to such a full-res…
▽ More
Accurately predicting the long-term behavior of chaotic systems is crucial for various applications such as climate modeling. However, achieving such predictions typically requires iterative computations over a dense spatiotemporal grid to account for the unstable nature of chaotic systems, which is expensive and impractical in many real-world situations. An alternative approach to such a full-resolved simulation is using a coarse grid and then correcting its errors through a \textit{closure model}, which approximates the overall information from fine scales not captured in the coarse-grid simulation. Recently, ML approaches have been used for closure modeling, but they typically require a large number of training samples from expensive fully-resolved simulations (FRS). In this work, we prove an even more fundamental limitation, i.e., the standard approach to learning closure models suffers from a large approximation error for generic problems, no matter how large the model is, and it stems from the non-uniqueness of the mapping. We propose an alternative end-to-end learning approach using a physics-informed neural operator (PINO) that overcomes this limitation by not using a closure model or a coarse-grid solver. We first train the PINO model on data from a coarse-grid solver and then fine-tune it with (a small amount of) FRS and physics-based losses on a fine grid. The discretization-free nature of neural operators means that they do not suffer from the restriction of a coarse grid that closure models face, and they can provably approximate the long-term statistics of chaotic systems. In our experiments, our PINO model achieves a 120x speedup compared to FRS with a relative error $\sim 5\%$. In contrast, the closure model coupled with a coarse-grid solver is $58$x slower than PINO while having a much higher error $\sim205\%$ when the closure model is trained on the same FRS dataset.
△ Less
Submitted 9 August, 2024;
originally announced August 2024.
-
Dynamical Measure Transport and Neural PDE Solvers for Sampling
Authors:
Jingtong Sun,
Julius Berner,
Lorenz Richter,
Marius Zeinhofer,
Johannes Müller,
Kamyar Azizzadenesheli,
Anima Anandkumar
Abstract:
The task of sampling from a probability density can be approached as transporting a tractable density function to the target, known as dynamical measure transport. In this work, we tackle it through a principled unified framework using deterministic or stochastic evolutions described by partial differential equations (PDEs). This framework incorporates prior trajectory-based sampling methods, such…
▽ More
The task of sampling from a probability density can be approached as transporting a tractable density function to the target, known as dynamical measure transport. In this work, we tackle it through a principled unified framework using deterministic or stochastic evolutions described by partial differential equations (PDEs). This framework incorporates prior trajectory-based sampling methods, such as diffusion models or Schrödinger bridges, without relying on the concept of time-reversals. Moreover, it allows us to propose novel numerical methods for solving the transport task and thus sampling from complicated targets without the need for the normalization constant or data samples. We employ physics-informed neural networks (PINNs) to approximate the respective PDE solutions, implying both conceptional and computational advantages. In particular, PINNs allow for simulation- and discretization-free optimization and can be trained very efficiently, leading to significantly better mode coverage in the sampling task compared to alternative methods. Moreover, they can readily be fine-tuned with Gauss-Newton methods to achieve high accuracy in sampling.
△ Less
Submitted 10 July, 2024;
originally announced July 2024.
-
Reduced-Order Neural Operators: Learning Lagrangian Dynamics on Highly Sparse Graphs
Authors:
Hrishikesh Viswanath,
Yue Chang,
Julius Berner,
Peter Yichen Chen,
Aniket Bera
Abstract:
We present a neural operator architecture to simulate Lagrangian dynamics, such as fluid flow, granular flows, and elastoplasticity. Traditional numerical methods, such as the finite element method (FEM), suffer from long run times and large memory consumption. On the other hand, approaches based on graph neural networks are faster but still suffer from long computation times on dense graphs, whic…
▽ More
We present a neural operator architecture to simulate Lagrangian dynamics, such as fluid flow, granular flows, and elastoplasticity. Traditional numerical methods, such as the finite element method (FEM), suffer from long run times and large memory consumption. On the other hand, approaches based on graph neural networks are faster but still suffer from long computation times on dense graphs, which are often required for high-fidelity simulations. Our model, GIOROM or Graph Interaction Operator for Reduced-Order Modeling, learns temporal dynamics within a reduced-order setting, capturing spatial features from a highly sparse graph representation of the input and generalizing to arbitrary spatial locations during inference. The model is geometry-aware and discretization-agnostic and can generalize to different initial conditions, velocities, and geometries after training. We show that point clouds of the order of 100,000 points can be inferred from sparse graphs with $\sim$1000 points, with negligible change in computation time. We empirically evaluate our model on elastic solids, Newtonian fluids, Non-Newtonian fluids, Drucker-Prager granular flows, and von Mises elastoplasticity. On these benchmarks, our approach results in a 25$\times$ speedup compared to other neural network-based physics simulators while delivering high-fidelity predictions of complex physical systems and showing better performance on most benchmarks. The code and the demos are provided at https://github.com/HrishikeshVish/GIOROM.
△ Less
Submitted 4 July, 2024;
originally announced July 2024.
-
Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing
Authors:
Bingliang Zhang,
Wenda Chu,
Julius Berner,
Chenlin Meng,
Anima Anandkumar,
Yang Song
Abstract:
Diffusion models have recently achieved success in solving Bayesian inverse problems with learned data priors. Current methods build on top of the diffusion sampling process, where each denoising step makes small modifications to samples from the previous step. However, this process struggles to correct errors from earlier sampling steps, leading to worse performance in complicated nonlinear inver…
▽ More
Diffusion models have recently achieved success in solving Bayesian inverse problems with learned data priors. Current methods build on top of the diffusion sampling process, where each denoising step makes small modifications to samples from the previous step. However, this process struggles to correct errors from earlier sampling steps, leading to worse performance in complicated nonlinear inverse problems, such as phase retrieval. To address this challenge, we propose a new method called Decoupled Annealing Posterior Sampling (DAPS) that relies on a novel noise annealing process. Specifically, we decouple consecutive steps in a diffusion sampling trajectory, allowing them to vary considerably from one another while ensuring their time-marginals anneal to the true posterior as we reduce noise levels. This approach enables the exploration of a larger solution space, improving the success rate for accurate reconstructions. We demonstrate that DAPS significantly improves sample quality and stability across multiple image restoration tasks, particularly in complicated nonlinear inverse problems. For example, we achieve a PSNR of 30.72dB on the FFHQ 256 dataset for phase retrieval, which is an improvement of 9.12dB compared to existing methods.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Solving Poisson Equations using Neural Walk-on-Spheres
Authors:
Hong Chul Nam,
Julius Berner,
Anima Anandkumar
Abstract:
We propose Neural Walk-on-Spheres (NWoS), a novel neural PDE solver for the efficient solution of high-dimensional Poisson equations. Leveraging stochastic representations and Walk-on-Spheres methods, we develop novel losses for neural networks based on the recursive solution of Poisson equations on spheres inside the domain. The resulting method is highly parallelizable and does not require spati…
▽ More
We propose Neural Walk-on-Spheres (NWoS), a novel neural PDE solver for the efficient solution of high-dimensional Poisson equations. Leveraging stochastic representations and Walk-on-Spheres methods, we develop novel losses for neural networks based on the recursive solution of Poisson equations on spheres inside the domain. The resulting method is highly parallelizable and does not require spatial gradients for the loss. We provide a comprehensive comparison against competing methods based on PINNs, the Deep Ritz method, and (backward) stochastic differential equations. In several challenging, high-dimensional numerical examples, we demonstrate the superiority of NWoS in accuracy, speed, and computational costs. Compared to commonly used PINNs, our approach can reduce memory usage and errors by orders of magnitude. Furthermore, we apply NWoS to problems in PDE-constrained optimization and molecular dynamics to show its efficiency in practical applications.
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs
Authors:
Md Ashiqur Rahman,
Robert Joseph George,
Mogab Elleithy,
Daniel Leibovici,
Zongyi Li,
Boris Bonev,
Colin White,
Julius Berner,
Raymond A. Yeh,
Jean Kossaifi,
Kamyar Azizzadenesheli,
Anima Anandkumar
Abstract:
Existing neural operator architectures face challenges when solving multiphysics problems with coupled partial differential equations (PDEs), due to complex geometries, interactions between physical variables, and the lack of large amounts of high-resolution training data. To address these issues, we propose Codomain Attention Neural Operator (CoDA-NO), which tokenizes functions along the codomain…
▽ More
Existing neural operator architectures face challenges when solving multiphysics problems with coupled partial differential equations (PDEs), due to complex geometries, interactions between physical variables, and the lack of large amounts of high-resolution training data. To address these issues, we propose Codomain Attention Neural Operator (CoDA-NO), which tokenizes functions along the codomain or channel space, enabling self-supervised learning or pretraining of multiple PDE systems. Specifically, we extend positional encoding, self-attention, and normalization layers to the function space. CoDA-NO can learn representations of different PDE systems with a single model. We evaluate CoDA-NO's potential as a backbone for learning multiphysics PDEs over multiple systems by considering few-shot learning settings. On complex downstream tasks with limited data, such as fluid flow simulations and fluid-structure interactions, we found CoDA-NO to outperform existing methods on the few-shot learning task by over $36\%$. The code is available at https://github.com/ashiq24/CoDA-NO.
△ Less
Submitted 5 April, 2024; v1 submitted 19 March, 2024;
originally announced March 2024.
-
DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
Authors:
Zhongkai Hao,
Chang Su,
Songming Liu,
Julius Berner,
Chengyang Ying,
Hang Su,
Anima Anandkumar,
Jian Song,
Jun Zhu
Abstract:
Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training s…
▽ More
Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at \url{https://github.com/thu-ml/DPOT}.
△ Less
Submitted 6 May, 2024; v1 submitted 6 March, 2024;
originally announced March 2024.
-
Neural Operators with Localized Integral and Differential Kernels
Authors:
Miguel Liu-Schiaffini,
Julius Berner,
Boris Bonev,
Thorsten Kurth,
Kamyar Azizzadenesheli,
Anima Anandkumar
Abstract:
Neural operators learn mappings between function spaces, which is practical for learning solution operators of PDEs and other scientific modeling applications. Among them, the Fourier neural operator (FNO) is a popular architecture that performs global convolutions in the Fourier space. However, such global operations are often prone to over-smoothing and may fail to capture local details. In cont…
▽ More
Neural operators learn mappings between function spaces, which is practical for learning solution operators of PDEs and other scientific modeling applications. Among them, the Fourier neural operator (FNO) is a popular architecture that performs global convolutions in the Fourier space. However, such global operations are often prone to over-smoothing and may fail to capture local details. In contrast, convolutional neural networks (CNN) can capture local features but are limited to training and inference at a single resolution. In this work, we present a principled approach to operator learning that can capture local features under two frameworks by learning differential operators and integral operators with locally supported kernels. Specifically, inspired by stencil methods, we prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions. Both these approaches preserve the properties of operator learning and, hence, the ability to predict at any resolution. Adding our layers to FNOs significantly improves their performance, reducing the relative L2-error by 34-72% in our experiments, which include a turbulent 2D Navier-Stokes and the spherical shallow water equations.
△ Less
Submitted 8 June, 2024; v1 submitted 26 February, 2024;
originally announced February 2024.
-
Large Language Models for Mathematicians
Authors:
Simon Frieder,
Julius Berner,
Philipp Petersen,
Thomas Lukasiewicz
Abstract:
Large language models (LLMs) such as ChatGPT have received immense interest for their general-purpose language understanding and, in particular, their ability to generate high-quality text or computer code. For many professions, LLMs represent an invaluable tool that can speed up and improve the quality of work. In this note, we discuss to what extent they can aid professional mathematicians. We f…
▽ More
Large language models (LLMs) such as ChatGPT have received immense interest for their general-purpose language understanding and, in particular, their ability to generate high-quality text or computer code. For many professions, LLMs represent an invaluable tool that can speed up and improve the quality of work. In this note, we discuss to what extent they can aid professional mathematicians. We first provide a mathematical description of the transformer model used in all modern language models. Based on recent studies, we then outline best practices and potential issues and report on the mathematical abilities of language models. Finally, we shed light on the potential of LLMs to change how mathematicians work.
△ Less
Submitted 2 April, 2024; v1 submitted 7 December, 2023;
originally announced December 2023.
-
Benefits of Deterministic and Stochastic Tendency Adjustments in a Climate Model
Authors:
William E. Chapman,
Judith Berner
Abstract:
We develop and compare model-error representation schemes derived from data assimilation increments and nudging tendencies in multi-decadal simulations of the community atmosphere model, version 6. Each scheme applies a bias correction during simulation run-time to the zonal and meridional winds. We quantify to which extent such online adjustment schemes improve the model climatology and variabili…
▽ More
We develop and compare model-error representation schemes derived from data assimilation increments and nudging tendencies in multi-decadal simulations of the community atmosphere model, version 6. Each scheme applies a bias correction during simulation run-time to the zonal and meridional winds. We quantify to which extent such online adjustment schemes improve the model climatology and variability on daily to seasonal timescales. Generally, we observe a ca. 30% improvement to annual upper-level zonal winds, with largest improvements in boreal spring (ca. 35%) and winter (ca. 47%). Despite only adjusting the wind fields, we additionally observe a ca. 20% improvement to annual precipitation over land, with the largest improvements in boreal fall (ca. 36%) and winter (ca. 25%), and a ca. 50% improvement to annual sea level pressure, globally. With mean state adjustments alone, the dominant pattern of boreal low-frequency variability over the Atlantic (the North Atlantic Oscillation) is significantly improved. Additional stochasticity further increases the modal explained variances, which brings it closer to the observed value. A streamfunction tendency decomposition reveals that the improvement is due to an adjustment to the high- and low-frequency eddy-eddy interaction terms. In the Pacific, the mean state adjustment alone led to an erroneous deepening of the Aleutian low, but this was remedied with the addition of stochastically selected tendencies. Finally, from a practical standpoint, we discuss the performance of using data assimilation increments versus nudging tendencies for an online model-error representation.
△ Less
Submitted 29 August, 2023;
originally announced August 2023.
-
Improved sampling via learned diffusions
Authors:
Lorenz Richter,
Julius Berner
Abstract:
Recently, a series of papers proposed deep learning-based approaches to sample from target distributions using controlled diffusion processes, being trained only on the unnormalized target densities without access to samples. Building on previous work, we identify these approaches as special cases of a generalized Schrödinger bridge problem, seeking a stochastic evolution between a given prior dis…
▽ More
Recently, a series of papers proposed deep learning-based approaches to sample from target distributions using controlled diffusion processes, being trained only on the unnormalized target densities without access to samples. Building on previous work, we identify these approaches as special cases of a generalized Schrödinger bridge problem, seeking a stochastic evolution between a given prior distribution and the specified target. We further generalize this framework by introducing a variational formulation based on divergences between path space measures of time-reversed diffusion processes. This abstract perspective leads to practical losses that can be optimized by gradient-based algorithms and includes previous objectives as special cases. At the same time, it allows us to consider divergences other than the reverse Kullback-Leibler divergence that is known to suffer from mode collapse. In particular, we propose the so-called log-variance loss, which exhibits favorable numerical properties and leads to significantly improved performance across all considered approaches.
△ Less
Submitted 23 May, 2024; v1 submitted 3 July, 2023;
originally announced July 2023.
-
Mathematical Capabilities of ChatGPT
Authors:
Simon Frieder,
Luca Pinchetti,
Alexis Chevalier,
Ryan-Rhys Griffiths,
Tommaso Salvatori,
Thomas Lukasiewicz,
Philipp Christian Petersen,
Julius Berner
Abstract:
We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-languag…
▽ More
We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!
△ Less
Submitted 20 July, 2023; v1 submitted 31 January, 2023;
originally announced January 2023.
-
An optimal control perspective on diffusion-based generative modeling
Authors:
Julius Berner,
Lorenz Richter,
Karen Ullrich
Abstract:
We establish a connection between stochastic optimal control and generative models based on stochastic differential equations (SDEs), such as recently developed diffusion probabilistic models. In particular, we derive a Hamilton-Jacobi-Bellman equation that governs the evolution of the log-densities of the underlying SDE marginals. This perspective allows to transfer methods from optimal control t…
▽ More
We establish a connection between stochastic optimal control and generative models based on stochastic differential equations (SDEs), such as recently developed diffusion probabilistic models. In particular, we derive a Hamilton-Jacobi-Bellman equation that governs the evolution of the log-densities of the underlying SDE marginals. This perspective allows to transfer methods from optimal control theory to generative modeling. First, we show that the evidence lower bound is a direct consequence of the well-known verification theorem from control theory. Further, we can formulate diffusion-based generative modeling as a minimization of the Kullback-Leibler divergence between suitable measures in path space. Finally, we develop a novel diffusion-based method for sampling from unnormalized densities -- a problem frequently occurring in statistics and computational sciences. We demonstrate that our time-reversed diffusion sampler (DIS) can outperform other diffusion-based sampling approaches on multiple numerical examples.
△ Less
Submitted 26 March, 2024; v1 submitted 2 November, 2022;
originally announced November 2022.
-
Robust SDE-Based Variational Formulations for Solving Linear PDEs via Deep Learning
Authors:
Lorenz Richter,
Julius Berner
Abstract:
The combination of Monte Carlo methods and deep learning has recently led to efficient algorithms for solving partial differential equations (PDEs) in high dimensions. Related learning problems are often stated as variational formulations based on associated stochastic differential equations (SDEs), which allow the minimization of corresponding losses using gradient-based optimization methods. In…
▽ More
The combination of Monte Carlo methods and deep learning has recently led to efficient algorithms for solving partial differential equations (PDEs) in high dimensions. Related learning problems are often stated as variational formulations based on associated stochastic differential equations (SDEs), which allow the minimization of corresponding losses using gradient-based optimization methods. In respective numerical implementations it is therefore crucial to rely on adequate gradient estimators that exhibit low variance in order to reach convergence accurately and swiftly. In this article, we rigorously investigate corresponding numerical aspects that appear in the context of linear Kolmogorov PDEs. In particular, we systematically compare existing deep learning approaches and provide theoretical explanations for their performances. Subsequently, we suggest novel methods that can be shown to be more robust both theoretically and numerically, leading to substantial performance improvements.
△ Less
Submitted 5 August, 2022; v1 submitted 21 June, 2022;
originally announced June 2022.
-
Learning ReLU networks to high uniform accuracy is intractable
Authors:
Julius Berner,
Philipp Grohs,
Felix Voigtlaender
Abstract:
Statistical learning theory provides bounds on the necessary number of training samples needed to reach a prescribed accuracy in a learning problem formulated over a given target class. This accuracy is typically measured in terms of a generalization error, that is, an expected value of a given loss function. However, for several applications -- for example in a security-critical context or for pr…
▽ More
Statistical learning theory provides bounds on the necessary number of training samples needed to reach a prescribed accuracy in a learning problem formulated over a given target class. This accuracy is typically measured in terms of a generalization error, that is, an expected value of a given loss function. However, for several applications -- for example in a security-critical context or for problems in the computational sciences -- accuracy in this sense is not sufficient. In such cases, one would like to have guarantees for high accuracy on every input value, that is, with respect to the uniform norm. In this paper we precisely quantify the number of training samples needed for any conceivable training algorithm to guarantee a given uniform accuracy on any learning problem formulated over target classes containing (or consisting of) ReLU neural networks of a prescribed architecture. We prove that, under very general assumptions, the minimal number of training samples for this task scales exponentially both in the depth and the input dimension of the network architecture.
△ Less
Submitted 28 February, 2023; v1 submitted 26 May, 2022;
originally announced May 2022.
-
The Modern Mathematics of Deep Learning
Authors:
Julius Berner,
Philipp Grohs,
Gitta Kutyniok,
Philipp Petersen
Abstract:
We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surpr…
▽ More
We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.
△ Less
Submitted 8 February, 2023; v1 submitted 9 May, 2021;
originally announced May 2021.
-
Two step micro-rheological behavior in a viscoelastic fluid
Authors:
Rohit Jain,
Félix Ginot,
Johannes Berner,
Clemens Bechinger,
Matthias Krüger
Abstract:
We perform micro-rheological experiments with a colloidal bead driven through a viscoelastic worm-like micellar fluid and observe two distinctive shear thinning regimes, each of them displaying a Newtonian-like plateau. The shear thinning behavior at larger velocities is in qualitative agreement with macroscopic rheological experiments. The second process, observed at Weissenberg numbers as small…
▽ More
We perform micro-rheological experiments with a colloidal bead driven through a viscoelastic worm-like micellar fluid and observe two distinctive shear thinning regimes, each of them displaying a Newtonian-like plateau. The shear thinning behavior at larger velocities is in qualitative agreement with macroscopic rheological experiments. The second process, observed at Weissenberg numbers as small as a few percent, appears to have no analog in macro rheological findings. A simple model introduced earlier captures the observed behavior, and implies that the two shear thinning processes correspond to two different length scales in the fluid. This model also reproduces oscillations which have been observed in this system previously. While the system under macro-shear seems to be near equilibrium for shear rates in the regime of the intermediate Newtonian-like plateau, the one under micro-shear is thus still far from it. The analysis suggests the existence of a length scale of a few micrometres, the nature of which remains elusive.
△ Less
Submitted 5 May, 2021; v1 submitted 22 February, 2021;
originally announced February 2021.
-
Numerically Solving Parametric Families of High-Dimensional Kolmogorov Partial Differential Equations via Deep Learning
Authors:
Julius Berner,
Markus Dablander,
Philipp Grohs
Abstract:
We present a deep learning algorithm for the numerical solution of parametric families of high-dimensional linear Kolmogorov partial differential equations (PDEs). Our method is based on reformulating the numerical approximation of a whole family of Kolmogorov PDEs as a single statistical learning problem using the Feynman-Kac formula. Successful numerical experiments are presented, which empirica…
▽ More
We present a deep learning algorithm for the numerical solution of parametric families of high-dimensional linear Kolmogorov partial differential equations (PDEs). Our method is based on reformulating the numerical approximation of a whole family of Kolmogorov PDEs as a single statistical learning problem using the Feynman-Kac formula. Successful numerical experiments are presented, which empirically confirm the functionality and efficiency of our proposed algorithm in the case of heat equations and Black-Scholes option pricing models parametrized by affine-linear coefficient functions. We show that a single deep neural network trained on simulated data is capable of learning the solution functions of an entire family of PDEs on a full space-time region. Most notably, our numerical observations and theoretical results also demonstrate that the proposed method does not suffer from the curse of dimensionality, distinguishing it from almost all standard numerical methods for PDEs.
△ Less
Submitted 9 November, 2020;
originally announced November 2020.
-
Properties of a nonlinear bath: Experiments, theory, and a stochastic Prandtl-Tomlinson model
Authors:
Boris Müller,
Johannes Berner,
Clemens Bechinger,
Matthias Krüger
Abstract:
A colloidal particle is a prominent example of a stochastic system, and, if suspended in a simple viscous liquid, very closely resembles the case of an ideal random walker. A variety of new phenomena have been observed when such colloid is suspended in a viscoelastic fluid instead, for example pronounced nonlinear responses when the viscoelastic bath is driven out of equilibrium. Here, using a mic…
▽ More
A colloidal particle is a prominent example of a stochastic system, and, if suspended in a simple viscous liquid, very closely resembles the case of an ideal random walker. A variety of new phenomena have been observed when such colloid is suspended in a viscoelastic fluid instead, for example pronounced nonlinear responses when the viscoelastic bath is driven out of equilibrium. Here, using a micron-sized particle in a micellar solution, we investigate in detail, how these nonlinear bath properties leave their fingerprints already in equilibrium measurements, for the cases where the particle is unconfined or trapped in a harmonic potential. We find that the coefficients in an effective linear (generalized) Langevin equation show intriguing inter-dependencies, which can be shown to arise only in nonlinear baths: For example, the friction memory can depend on the external potential that acts only on the colloidal particle (as recently noted in simulations of molecular tracers in water in Phys. Rev. X 7, 041065 (2017)), it can depend on the mass of the colloid, or, in an overdamped setting, on its bare diffusivity. These inter-dependencies, caused by so-called fluctuation renormalizations, are seen in an exact small time expansion of the friction memory based on microscopic starting points. Using linear response theory, they can be interpreted in terms of microrheological modes of force-controlled or velocity-controlled driving. The mentioned nonlinear markers are observed in our experiments, which are astonishingly well reproduced by a stochastic Prandtl-Tomlinson model mimicking the nonlinear viscoelastic bath. The pronounced nonlinearities seen in our experiments together with the good understanding in a simple theoretical model make this system a promising candidate for exploration of colloidal motion in nonlinear stochastic environments.
△ Less
Submitted 30 September, 2019; v1 submitted 27 September, 2019;
originally announced September 2019.
-
How degenerate is the parametrization of neural networks with the ReLU activation function?
Authors:
Julius Berner,
Dennis Elbrächter,
Philipp Grohs
Abstract:
Neural network training is usually accomplished by solving a non-convex optimization problem using stochastic gradient descent. Although one optimizes over the networks parameters, the main loss function generally only depends on the realization of the neural network, i.e. the function it computes. Studying the optimization problem over the space of realizations opens up new ways to understand neu…
▽ More
Neural network training is usually accomplished by solving a non-convex optimization problem using stochastic gradient descent. Although one optimizes over the networks parameters, the main loss function generally only depends on the realization of the neural network, i.e. the function it computes. Studying the optimization problem over the space of realizations opens up new ways to understand neural network training. In particular, usual loss functions like mean squared error and categorical cross entropy are convex on spaces of neural network realizations, which themselves are non-convex. Approximation capabilities of neural networks can be used to deal with the latter non-convexity, which allows us to establish that for sufficiently large networks local minima of a regularized optimization problem on the realization space are almost optimal. Note, however, that each realization has many different, possibly degenerate, parametrizations. In particular, a local minimum in the parametrization space needs not correspond to a local minimum in the realization space. To establish such a connection, inverse stability of the realization map is required, meaning that proximity of realizations must imply proximity of corresponding parametrizations. We present pathologies which prevent inverse stability in general, and, for shallow networks, proceed to establish a restricted space of parametrizations on which we have inverse stability w.r.t. to a Sobolev norm. Furthermore, we show that by optimizing over such restricted sets, it is still possible to learn any function which can be learned by optimization over unrestricted sets.
△ Less
Submitted 8 February, 2023; v1 submitted 23 May, 2019;
originally announced May 2019.
-
Towards a regularity theory for ReLU networks -- chain rule and global error estimates
Authors:
Julius Berner,
Dennis Elbrächter,
Philipp Grohs,
Arnulf Jentzen
Abstract:
Although for neural networks with locally Lipschitz continuous activation functions the classical derivative exists almost everywhere, the standard chain rule is in general not applicable. We will consider a way of introducing a derivative for neural networks that admits a chain rule, which is both rigorous and easy to work with. In addition we will present a method of converting approximation res…
▽ More
Although for neural networks with locally Lipschitz continuous activation functions the classical derivative exists almost everywhere, the standard chain rule is in general not applicable. We will consider a way of introducing a derivative for neural networks that admits a chain rule, which is both rigorous and easy to work with. In addition we will present a method of converting approximation results on bounded domains to global (pointwise) estimates. This can be used to extend known neural network approximation theory to include the study of regularity properties. Of particular interest is the application to neural networks with ReLU activation function, where it contributes to the understanding of the success of deep learning methods for high-dimensional partial differential equations.
△ Less
Submitted 13 May, 2019;
originally announced May 2019.
-
Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations
Authors:
Julius Berner,
Philipp Grohs,
Arnulf Jentzen
Abstract:
The development of new classification and regression algorithms based on empirical risk minimization (ERM) over deep neural network hypothesis classes, coined deep learning, revolutionized the area of artificial intelligence, machine learning, and data analysis. In particular, these methods have been applied to the numerical solution of high-dimensional partial differential equations with great su…
▽ More
The development of new classification and regression algorithms based on empirical risk minimization (ERM) over deep neural network hypothesis classes, coined deep learning, revolutionized the area of artificial intelligence, machine learning, and data analysis. In particular, these methods have been applied to the numerical solution of high-dimensional partial differential equations with great success. Recent simulations indicate that deep learning-based algorithms are capable of overcoming the curse of dimensionality for the numerical solution of Kolmogorov equations, which are widely used in models from engineering, finance, and the natural sciences. The present paper considers under which conditions ERM over a deep neural network hypothesis class approximates the solution of a $d$-dimensional Kolmogorov equation with affine drift and diffusion coefficients and typical initial values arising from problems in computational finance up to error $\varepsilon$. We establish that, with high probability over draws of training samples, such an approximation can be achieved with both the size of the hypothesis class and the number of training samples scaling only polynomially in $d$ and $\varepsilon^{-1}$. It can be concluded that ERM over deep neural network hypothesis classes overcomes the curse of dimensionality for the numerical solution of linear Kolmogorov equations with affine coefficients.
△ Less
Submitted 11 November, 2020; v1 submitted 9 September, 2018;
originally announced September 2018.
-
Oscillating Modes of Driven Colloids in Overdamped Systems
Authors:
Johannes Berner,
Boris Müller,
Juan Ruben Gomez-Solano,
Matthias Krüger,
Clemens Bechinger
Abstract:
Microscopic particles suspended in liquids are the prime example of an overdamped system because viscous forces dominate over inertial effects. Apart from their use as model systems, they receive considerable attention as sensitive probes from which forces on molecular scales can be inferred. The interpretation of such experiments rests on the assumption, that, even if the particles are driven, th…
▽ More
Microscopic particles suspended in liquids are the prime example of an overdamped system because viscous forces dominate over inertial effects. Apart from their use as model systems, they receive considerable attention as sensitive probes from which forces on molecular scales can be inferred. The interpretation of such experiments rests on the assumption, that, even if the particles are driven, the liquid remains in equilibrium, and all modes are overdamped. Here, we experimentally demonstrate that this is no longer valid when a particle is forced through a viscoelastic fluid. Even at small driving velocities where Stokes law remains valid, we observe particle oscillations with periods up to several tens of seconds. We attribute these to non-equilibrium fluctuations of the fluid, which are excited by the particle's motion. The observed oscillatory dynamics is in quantitative agreement with an overdamped Langevin equation with negative friction-memory term and which is equivalent to the motion of a stochastically driven underdamped oscillator. This fundamentally new oscillatory mode will largely expand the variety of model systems but has also considerable implications on how molecular forces are determined by colloidal probe particles under natural viscoelastic conditions.
△ Less
Submitted 3 February, 2018;
originally announced February 2018.
-
Etale homotopy theory of non-archimedean analytic spaces
Authors:
Joe Berner
Abstract:
We review the shape theory of $\infty$-topoi, and relate it with the usual cohomology of locally constant sheaves. Additionally, a new localization of profinite spaces is defined which allows us to extend the étale realization functor of Isaksen. We apply these ideas to define an étale homotopy type functor $\mathrm{\acute{e}t}(\mathcal{X})$ for Berkovich's non-archimedean analytic spaces…
▽ More
We review the shape theory of $\infty$-topoi, and relate it with the usual cohomology of locally constant sheaves. Additionally, a new localization of profinite spaces is defined which allows us to extend the étale realization functor of Isaksen. We apply these ideas to define an étale homotopy type functor $\mathrm{\acute{e}t}(\mathcal{X})$ for Berkovich's non-archimedean analytic spaces $\mathcal{X}$ over a complete non-archimedean field $K$ and prove some properties of the construction. We compare the étale homotopy types coming from Tate's rigid spaces, Huber's adic spaces, and rigid models when they are all defined.
△ Less
Submitted 11 August, 2017;
originally announced August 2017.
-
Stochastic Parameterization: Towards a new view of Weather and Climate Models
Authors:
Judith Berner,
Ulrich Achatz,
Lauriane Batte,
Lisa Bengtsson,
Alvaro De La Camara,
Daan Crommelin,
Hannah Christensen,
Matteo Colangeli,
Stamen Dolaptchiev,
Christian L. E. Franzke,
Petra Friederichs,
Peter Imkeller,
Heikki Jarvinen,
Stephan Juricke,
Vassili Kitsios,
Franois Lott,
Valerio Lucarini,
Salil Mahajan,
Timothy N. Palmer,
Cecile Penland,
Jin-Song Von Storch,
Mirjana Sakradzija,
Michael Weniger,
Antje Weisheimer,
Paul D. Williams
, et al. (1 additional authors not shown)
Abstract:
The last decade has seen the success of stochastic parameterizations in short-term, medium-range and seasonal forecasts: operational weather centers now routinely use stochastic parameterization schemes to better represent model inadequacy and improve the quantification of forecast uncertainty. Developed initially for numerical weather prediction, the inclusion of stochastic parameterizations not…
▽ More
The last decade has seen the success of stochastic parameterizations in short-term, medium-range and seasonal forecasts: operational weather centers now routinely use stochastic parameterization schemes to better represent model inadequacy and improve the quantification of forecast uncertainty. Developed initially for numerical weather prediction, the inclusion of stochastic parameterizations not only provides better estimates of uncertainty, but it is also extremely promising for reducing longstanding climate biases and relevant for determining the climate response to external forcing. This article highlights recent developments from different research groups which show that the stochastic representation of unresolved processes in the atmosphere, oceans, land surface and cryosphere of comprehensive weather and climate models (a) gives rise to more reliable probabilistic forecasts of weather and climate and (b) reduces systematic model bias. We make a case that the use of mathematically stringent methods for the derivation of stochastic dynamic equations will lead to substantial improvements in our ability to accurately simulate weather and climate at all scales. Recent work in mathematics, statistical mechanics and turbulence is reviewed, its relevance for the climate problem demonstrated, and future research directions outlined.
△ Less
Submitted 20 March, 2017; v1 submitted 29 October, 2015;
originally announced October 2015.
-
Stochastic Climate Theory and Modelling
Authors:
Christian L. E. Franzke,
Terence J. O'Kane,
Judith Berner,
Paul D. Williams,
Valerio Lucarini
Abstract:
Stochastic methods are a crucial area in contemporary climate research and are increasingly being used in comprehensive weather and climate prediction models as well as reduced order climate models. Stochastic methods are used as subgrid-scale parameterizations as well as for model error representation, uncertainty quantification, data assimilation and ensemble prediction. The need to use stochast…
▽ More
Stochastic methods are a crucial area in contemporary climate research and are increasingly being used in comprehensive weather and climate prediction models as well as reduced order climate models. Stochastic methods are used as subgrid-scale parameterizations as well as for model error representation, uncertainty quantification, data assimilation and ensemble prediction. The need to use stochastic approaches in weather and climate models arises because we still cannot resolve all necessary processes and scales in comprehensive numerical weather and climate prediction models. In many practical applications one is mainly interested in the largest and potentially predictable scales and not necessarily in the small and fast scales. For instance, reduced order models can simulate and predict large scale modes. Statistical mechanics and dynamical systems theory suggest that in reduced order models the impact of unresolved degrees of freedom can be represented by suitable combinations of deterministic and stochastic components and non-Markovian (memory) terms. Stochastic approaches in numerical weather and climate prediction models also lead to the reduction of model biases. Hence, there is a clear need for systematic stochastic approaches in weather and climate modelling. In this review we present evidence for stochastic effects in laboratory experiments. Then we provide an overview of stochastic climate theory from an applied mathematics perspectives. We also survey the current use of stochastic methods in comprehensive weather and climate prediction models and show that stochastic parameterizations have the potential to remedy many of the current biases in these comprehensive models.
△ Less
Submitted 1 September, 2014;
originally announced September 2014.
-
Towards the Probabilistic Earth-System Model
Authors:
T. N. Palmer,
F. J. Doblas-Reyes,
A. Weisheimer,
G. J. Shutts,
J. Berner,
J. M. Murphy
Abstract:
Multi-model ensembles provide a pragmatic approach to the representation of model uncertainty in climate prediction. However, such representations are inherently ad hoc, and, as shown, probability distributions of climate variables based on current-generation multi-model ensembles, are not accurate. Results from seasonal re-forecast studies suggest that climate model ensembles based on stochasti…
▽ More
Multi-model ensembles provide a pragmatic approach to the representation of model uncertainty in climate prediction. However, such representations are inherently ad hoc, and, as shown, probability distributions of climate variables based on current-generation multi-model ensembles, are not accurate. Results from seasonal re-forecast studies suggest that climate model ensembles based on stochastic-dynamic parametrisation are beginning to outperform multi-model ensembles, and have the potential to become significantly more skilful than multi-model ensembles.
The case is made for stochastic representations of model uncertainty in future-generation climate prediction models. Firstly, a guiding characteristic of the scientific method is an ability to characterise and predict uncertainty; individual climate models are not currently able to do this. Secondly, through the effects of noise-induced rectification, stochastic-dynamic parametrisation may provide a (poor man's) surrogate to high resolution. Thirdly, stochastic-dynamic parametrisations may be able to take advantage of the inherent stochasticity of electron flow through certain types of low-energy computer chips, currently under development.
These arguments have particular resonance for next-generation Earth-System models, which purport to be comprehensive numerical representations of climate, and where integrations at high resolution may be unaffordable.
△ Less
Submitted 26 August, 2009; v1 submitted 5 December, 2008;
originally announced December 2008.