-
Sparse $L^1$-Autoencoders for Scientific Data Compression
Authors:
Matthias Chung,
Rick Archibald,
Paul Atzberger,
Jack Michael Solomon
Abstract:
Scientific datasets present unique challenges for machine learning-driven compression methods, including more stringent requirements on accuracy and mitigation of potential invalidating artifacts. Drawing on results from compressed sensing and rate-distortion theory, we introduce effective data compression methods by developing autoencoders using high dimensional latent spaces that are $L^1$-regul…
▽ More
Scientific datasets present unique challenges for machine learning-driven compression methods, including more stringent requirements on accuracy and mitigation of potential invalidating artifacts. Drawing on results from compressed sensing and rate-distortion theory, we introduce effective data compression methods by developing autoencoders using high dimensional latent spaces that are $L^1$-regularized to obtain sparse low dimensional representations. We show how these information-rich latent spaces can be used to mitigate blurring and other artifacts to obtain highly effective data compression methods for scientific data. We demonstrate our methods for short angle scattering (SAS) datasets showing they can achieve compression ratios around two orders of magnitude and in some cases better. Our compression methods show promise for use in addressing current bottlenecks in transmission, storage, and analysis in high-performance distributed computing environments. This is central to processing the large volume of SAS data being generated at shared experimental facilities around the world to support scientific investigations. Our approaches provide general ways for obtaining specialized compression methods for targeted scientific datasets.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Paired Autoencoders for Inverse Problems
Authors:
Matthias Chung,
Emma Hart,
Julianne Chung,
Bas Peters,
Eldad Haber
Abstract:
We consider the solution of nonlinear inverse problems where the forward problem is a discretization of a partial differential equation. Such problems are notoriously difficult to solve in practice and require minimizing a combination of a data-fit term and a regularization term. The main computational bottleneck of typical algorithms is the direct estimation of the data misfit. Therefore, likelih…
▽ More
We consider the solution of nonlinear inverse problems where the forward problem is a discretization of a partial differential equation. Such problems are notoriously difficult to solve in practice and require minimizing a combination of a data-fit term and a regularization term. The main computational bottleneck of typical algorithms is the direct estimation of the data misfit. Therefore, likelihood-free approaches have become appealing alternatives. Nonetheless, difficulties in generalization and limitations in accuracy have hindered their broader utility and applicability. In this work, we use a paired autoencoder framework as a likelihood-free estimator for inverse problems. We show that the use of such an architecture allows us to construct a solution efficiently and to overcome some known open problems when using likelihood-free estimators. In particular, our framework can assess the quality of the solution and improve on it if needed. We demonstrate the viability of our approach using examples from full waveform inversion and inverse electromagnetic imaging.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
Image reconstructions using sparse dictionary representations and implicit, non-negative mappings
Authors:
Elizabeth Newman,
Jack Michael Solomon,
Matthias Chung
Abstract:
Many imaging science tasks can be modeled as a discrete linear inverse problem. Solving linear inverse problems is often challenging, with ill-conditioned operators and potentially non-unique solutions. Embedding prior knowledge, such as smoothness, into the solution can overcome these challenges. In this work, we encode prior knowledge using a non-negative patch dictionary, which effectively lear…
▽ More
Many imaging science tasks can be modeled as a discrete linear inverse problem. Solving linear inverse problems is often challenging, with ill-conditioned operators and potentially non-unique solutions. Embedding prior knowledge, such as smoothness, into the solution can overcome these challenges. In this work, we encode prior knowledge using a non-negative patch dictionary, which effectively learns a basis from a training set of natural images. In this dictionary basis, we desire solutions that are non-negative and sparse (i.e., contain many zero entries). With these constraints, standard methods for solving discrete linear inverse problems are not directly applicable. One such approach is the modified residual norm steepest descent (MRNSD), which produces non-negative solutions but does not induce sparsity. In this paper, we provide two methods based on MRNSD that promote sparsity. In our first method, we add an $\ell_1$-regularization term with a new, optimal step size. In our second method, we propose a new non-negative, sparsity-promoting mapping of the solution. We compare the performance of our proposed methods on a number of numerical experiments, including deblurring, image completion, computer tomography, and superresolution. Our results show that these methods effectively solve discrete linear inverse problems with non-negativity and sparsity constraints.
△ Less
Submitted 5 December, 2023;
originally announced December 2023.
-
Goal-oriented Uncertainty Quantification for Inverse Problems via Variational Encoder-Decoder Networks
Authors:
Babak Maboudi Afkham,
Julianne Chung,
Matthias Chung
Abstract:
In this work, we describe a new approach that uses variational encoder-decoder (VED) networks for efficient goal-oriented uncertainty quantification for inverse problems. Contrary to standard inverse problems, these approaches are \emph{goal-oriented} in that the goal is to estimate some quantities of interest (QoI) that are functions of the solution of an inverse problem, rather than the solution…
▽ More
In this work, we describe a new approach that uses variational encoder-decoder (VED) networks for efficient goal-oriented uncertainty quantification for inverse problems. Contrary to standard inverse problems, these approaches are \emph{goal-oriented} in that the goal is to estimate some quantities of interest (QoI) that are functions of the solution of an inverse problem, rather than the solution itself. Moreover, we are interested in computing uncertainty metrics associated with the QoI, thus utilizing a Bayesian approach for inverse problems that incorporates the prediction operator and techniques for exploring the posterior. This may be particularly challenging, especially for nonlinear, possibly unknown, operators and nonstandard prior assumptions. We harness recent advances in machine learning, i.e., VED networks, to describe a data-driven approach to large-scale inverse problems. This enables a real-time goal-oriented uncertainty quantification for the QoI. One of the advantages of our approach is that we avoid the need to solve challenging inversion problems by training a network to approximate the mapping from observations to QoI. Another main benefit is that we enable uncertainty quantification for the QoI by leveraging probability distributions in the latent space. This allows us to efficiently generate QoI samples and circumvent complicated or even unknown forward models and prediction operators. Numerical results from medical tomography reconstruction and nonlinear hydraulic tomography demonstrate the potential and broad applicability of the approach.
△ Less
Submitted 29 September, 2023; v1 submitted 17 April, 2023;
originally announced April 2023.
-
PH-STAT
Authors:
Moo K. Chung
Abstract:
We introduce PH-STAT, a comprehensive Matlab toolbox designed for performing a wide range of statistical inferences on persistent homology. Persistent homology is a prominent tool in topological data analysis (TDA) that captures the underlying topological features of complex data sets. The toolbox aims to provide users with an accessible and user-friendly interface for analyzing and interpreting t…
▽ More
We introduce PH-STAT, a comprehensive Matlab toolbox designed for performing a wide range of statistical inferences on persistent homology. Persistent homology is a prominent tool in topological data analysis (TDA) that captures the underlying topological features of complex data sets. The toolbox aims to provide users with an accessible and user-friendly interface for analyzing and interpreting topological data. The package is distributed in https://github.com/laplcebeltrami/PH-STAT.
△ Less
Submitted 12 April, 2023;
originally announced April 2023.
-
The Variable Projected Augmented Lagrangian Method
Authors:
Matthias Chung,
Rosemary Renaut
Abstract:
Inference by means of mathematical modeling from a collection of observations remains a crucial tool for scientific discovery and is ubiquitous in application areas such as signal compression, imaging restoration, and supervised machine learning. The inference problems may be solved using variational formulations that provide theoretically proven methods and algorithms. With ever-increasing model…
▽ More
Inference by means of mathematical modeling from a collection of observations remains a crucial tool for scientific discovery and is ubiquitous in application areas such as signal compression, imaging restoration, and supervised machine learning. The inference problems may be solved using variational formulations that provide theoretically proven methods and algorithms. With ever-increasing model complexities and growing data size, new specially designed methods are urgently needed to recover meaningful quantifies of interest. We consider the broad spectrum of linear inverse problems where the aim is to reconstruct quantities with a sparse representation on some vector space; often solved using the (generalized) least absolute shrinkage and selection operator (lasso). The associated optimization problems have received significant attention, in particular in the early 2000's, because of their connection to compressed sensing and the reconstruction of solutions with favorable sparsity properties using augmented Lagrangians, alternating directions and splitting methods. We provide a new perspective on the underlying l1 regularized inverse problem by exploring the generalized lasso problem through variable projection methods. We arrive at our proposed variable projected augmented Lagrangian (vpal) method. We analyze this method and provide an approach for automatic regularization parameter selection based on a degrees of freedom argument. Further, we provide numerical examples demonstrating the computational efficiency for various imaging problems.
△ Less
Submitted 17 July, 2022;
originally announced July 2022.
-
Unconstrained optimization using the directional proximal point method
Authors:
Ming-Yu Chung,
Jinn Ho,
Wen-Liang Hwang
Abstract:
This paper presents a directional proximal point method (DPPM) to derive the minimum of any C1-smooth function f. The proposed method requires a function persistent a local convex segment along the descent direction at any non-critical point (referred to a DLC direction at the point). The proposed DPPM can determine a DLC direction by solving a two-dimensional quadratic optimization problem, regar…
▽ More
This paper presents a directional proximal point method (DPPM) to derive the minimum of any C1-smooth function f. The proposed method requires a function persistent a local convex segment along the descent direction at any non-critical point (referred to a DLC direction at the point). The proposed DPPM can determine a DLC direction by solving a two-dimensional quadratic optimization problem, regardless of the dimensionally of the function variables. Along that direction, the DPPM then updates by solving a one-dimensional optimization problem. This gives the DPPM advantage over competitive methods when dealing with large-scale problems, involving a large number of variables. We show that the DPPM converges to critical points of f. We also provide conditions under which the entire DPPM sequence converges to a single critical point. For strongly convex quadratic functions, we demonstrate that the rate at which the error sequence converges to zero can be R-superlinear, regardless of the dimension of variables.
△ Less
Submitted 28 April, 2022;
originally announced April 2022.
-
Persistent Homological State-Space Estimation of Functional Human Brain Networks at Rest
Authors:
Moo K. Chung,
Shih-Gu Huang,
Ian C. Carroll,
Vince D. Calhoun,
H. Hill Goldsmith
Abstract:
We introduce an innovative, data-driven topological data analysis (TDA) technique for estimating the state spaces of dynamically changing functional human brain networks at rest. Our method utilizes the Wasserstein distance to measure topological differences, enabling the clustering of brain networks into distinct topological states. This technique outperforms the commonly used k-means clustering…
▽ More
We introduce an innovative, data-driven topological data analysis (TDA) technique for estimating the state spaces of dynamically changing functional human brain networks at rest. Our method utilizes the Wasserstein distance to measure topological differences, enabling the clustering of brain networks into distinct topological states. This technique outperforms the commonly used k-means clustering in identifying brain network state spaces by effectively incorporating the temporal dynamics of the data without the need for explicit model specification. We further investigate the genetic underpinnings of these topological features using a twin study design, examining the heritability of such state changes. Our findings suggest that the topology of brain networks, particularly in their dynamic state changes, may hold significant hidden genetic information. MATLAB code for the method is available at https://github.com/laplcebeltrami/PH-STAT.
△ Less
Submitted 16 April, 2024; v1 submitted 31 December, 2021;
originally announced January 2022.
-
Efficient learning methods for large-scale optimal inversion design
Authors:
Julianne Chung,
Matthias Chung,
Silvia Gazzola,
Mirjeta Pasha
Abstract:
In this work, we investigate various approaches that use learning from training data to solve inverse problems, following a bi-level learning approach. We consider a general framework for optimal inversion design, where training data can be used to learn optimal regularization parameters, data fidelity terms, and regularizers, thereby resulting in superior variational regularization methods. In pa…
▽ More
In this work, we investigate various approaches that use learning from training data to solve inverse problems, following a bi-level learning approach. We consider a general framework for optimal inversion design, where training data can be used to learn optimal regularization parameters, data fidelity terms, and regularizers, thereby resulting in superior variational regularization methods. In particular, we describe methods to learn optimal $p$ and $q$ norms for ${\rm L}^p-{\rm L}^q$ regularization and methods to learn optimal parameters for regularization matrices defined by covariance kernels. We exploit efficient algorithms based on Krylov projection methods for solving the regularized problems, both at training and validation stages, making these methods well-suited for large-scale problems. Our experiments show that the learned regularization methods perform well even when there is some inexactness in the forward operator, resulting in a mixture of model and measurement error.
△ Less
Submitted 6 October, 2021;
originally announced October 2021.
-
Least-Squares Finite Element Method for Ordinary Differential Equations
Authors:
Matthias Chung,
Justin Krueger,
Honghu Liu
Abstract:
We consider the least-squares finite element method (lsfem) for systems of nonlinear ordinary differential equations and establish an optimal error estimate for this method when piecewise linear elements are used. The main assumptions are that the vector field is sufficiently smooth and that the local Lipschitz constant, as well as the operator norm of the Jacobian matrix associated with the nonli…
▽ More
We consider the least-squares finite element method (lsfem) for systems of nonlinear ordinary differential equations and establish an optimal error estimate for this method when piecewise linear elements are used. The main assumptions are that the vector field is sufficiently smooth and that the local Lipschitz constant, as well as the operator norm of the Jacobian matrix associated with the nonlinearity, are sufficiently small when restricted to a suitable neighborhood of the true solution for the considered initial value problem. This theoretic optimality is further illustrated numerically, along with evidence of possible extension to higher-order basis elements. Examples are also presented to show the advantages of lsfem compared with finite difference methods in various scenarios. Suitable modifications for adaptive time-stepping are discussed as well.
△ Less
Submitted 30 September, 2021;
originally announced September 2021.
-
slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks
Authors:
Elizabeth Newman,
Julianne Chung,
Matthias Chung,
Lars Ruthotto
Abstract:
Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging in general. DNN training is commonly phrased as a stochastic optimization problem whose challenges include non-convexity, non-smoothness, insufficient regularization, and complicated data distributions. Hence, the performance of DNNs on a g…
▽ More
Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging in general. DNN training is commonly phrased as a stochastic optimization problem whose challenges include non-convexity, non-smoothness, insufficient regularization, and complicated data distributions. Hence, the performance of DNNs on a given task depends crucially on tuning hyperparameters, especially learning rates and regularization parameters. In the absence of theoretical guidelines or prior experience on similar tasks, this requires solving many training problems, which can be time-consuming and demanding on computational resources. This can limit the applicability of DNNs to problems with non-standard, complex, and scarce datasets, e.g., those arising in many scientific applications. To remedy the challenges of DNN training, we propose slimTrain, a stochastic optimization method for training DNNs with reduced sensitivity to the choice hyperparameters and fast initial convergence. The central idea of slimTrain is to exploit the separability inherent in many DNN architectures; that is, we separate the DNN into a nonlinear feature extractor followed by a linear model. This separability allows us to leverage recent advances made for solving large-scale, linear, ill-posed inverse problems. Crucially, for the linear weights, slimTrain does not require a learning rate and automatically adapts the regularization parameter. Since our method operates on mini-batches, its computational overhead per iteration is modest. In our numerical experiments, slimTrain outperforms existing DNN training methods with the recommended hyperparameter settings and reduces the sensitivity of DNN training to the remaining hyperparameters.
△ Less
Submitted 28 September, 2021;
originally announced September 2021.
-
Learning Regularization Parameters of Inverse Problems via Deep Neural Networks
Authors:
Babak Maboudi Afkham,
Julianne Chung,
Matthias Chung
Abstract:
In this work, we describe a new approach that uses deep neural networks (DNN) to obtain regularization parameters for solving inverse problems. We consider a supervised learning approach, where a network is trained to approximate the mapping from observation data to regularization parameters. Once the network is trained, regularization parameters for newly obtained data can be computed by efficien…
▽ More
In this work, we describe a new approach that uses deep neural networks (DNN) to obtain regularization parameters for solving inverse problems. We consider a supervised learning approach, where a network is trained to approximate the mapping from observation data to regularization parameters. Once the network is trained, regularization parameters for newly obtained data can be computed by efficient forward propagation of the DNN. We show that a wide variety of regularization functionals, forward models, and noise models may be considered. The network-obtained regularization parameters can be computed more efficiently and may even lead to more accurate solutions compared to existing regularization parameter selection methods. We emphasize that the key advantage of using DNNs for learning regularization parameters, compared to previous works on learning via optimal experimental design or empirical Bayes risk minimization, is greater generalizability. That is, rather than computing one set of parameters that is optimal with respect to one particular design objective, DNN-computed regularization parameters are tailored to the specific features or properties of the newly observed data. Thus, our approach may better handle cases where the observation is not a close representation of the training set. Furthermore, we avoid the need for expensive and challenging bilevel optimization methods as utilized in other existing training approaches. Numerical results demonstrate the potential of using DNNs to learn regularization parameters.
△ Less
Submitted 13 April, 2021;
originally announced April 2021.
-
Quenched limit theorems for random U(1) extensions of expanding maps
Authors:
Yong Moo Chung,
Yushi Nakano,
Jens Wittsten
Abstract:
The Lyapunov spectra of random U(1) extensions of expanding maps on the torus were investigated in our previous work [NW2015]. Using the result, we extend the recent spectral approach for quenched limit theorems for expanding maps [DFGV2018] and hyperbolic maps [DFGV2019] to our partially hyperbolic dynamics. Quenched central limit theorems, large deviations principles and local central limit theo…
▽ More
The Lyapunov spectra of random U(1) extensions of expanding maps on the torus were investigated in our previous work [NW2015]. Using the result, we extend the recent spectral approach for quenched limit theorems for expanding maps [DFGV2018] and hyperbolic maps [DFGV2019] to our partially hyperbolic dynamics. Quenched central limit theorems, large deviations principles and local central limit theorems for random U(1) extensions of expanding maps on the torus are proved via corresponding theorems for abstract random dynamical systems.
△ Less
Submitted 3 October, 2022; v1 submitted 4 April, 2021;
originally announced April 2021.
-
Reviews: Topological Distances and Losses for Brain Networks
Authors:
Moo K. Chung,
Alexander Smith,
Gary Shiu
Abstract:
Almost all statistical and machine learning methods in analyzing brain networks rely on distances and loss functions, which are mostly Euclidean or matrix norms. The Euclidean or matrix distances may fail to capture underlying subtle topological differences in brain networks. Further, Euclidean distances are sensitive to outliers. A few extreme edge weights may severely affect the distance. Thus i…
▽ More
Almost all statistical and machine learning methods in analyzing brain networks rely on distances and loss functions, which are mostly Euclidean or matrix norms. The Euclidean or matrix distances may fail to capture underlying subtle topological differences in brain networks. Further, Euclidean distances are sensitive to outliers. A few extreme edge weights may severely affect the distance. Thus it is necessary to use distances and loss functions that recognize topology of data. In this review paper, we survey various topological distance and loss functions from topological data analysis (TDA) and persistent homology that can be used in brain network analysis more effectively. Although there are many recent brain imaging studies that are based on TDA methods, possibly due to the lack of method awareness, TDA has not taken as the mainstream tool in brain imaging field yet. The main purpose of this paper is provide the relevant technical survey of these powerful tools that are immediately applicable to brain network data.
△ Less
Submitted 17 February, 2021;
originally announced February 2021.
-
Diffusion Equations for Medical Images
Authors:
Moo K. Chung
Abstract:
In brain imaging, the image acquisition and processing processes themselves are likely to introduce noise to the images. It is therefore imperative to reduce the noise while preserving the geometric details of the anatomical structures for various applications. Traditionally Gaussian kernel smoothing has been often used in brain image processing and analysis. However, the direct application of Gau…
▽ More
In brain imaging, the image acquisition and processing processes themselves are likely to introduce noise to the images. It is therefore imperative to reduce the noise while preserving the geometric details of the anatomical structures for various applications. Traditionally Gaussian kernel smoothing has been often used in brain image processing and analysis. However, the direct application of Gaussian kernel smoothing tend to cause various numerical issues in irregular domains with boundaries. For example, if one uses large bandwidth in kernel smoothing in a cortical bounded region, the smoothing will blur signals across boundaries. So in kernel smoothing and regression literature, various ad-hoc procedures were introduce to remedy the boundary effect. Diffusion equations have been widely used in brain imaging as a form of noise reduction. The most natural straightforward way to smooth images in irregular domains with boundaries is to formulate the problem as boundary value problems using partial differential equations. Numerous diffusion-based techniques have been developed in image processing. In this paper, we will overview the basics of isotropic diffusion equations and explain how to solve them on regular grids and irregular grids such as graphs.
△ Less
Submitted 2 January, 2022; v1 submitted 27 July, 2020;
originally announced July 2020.
-
Introduction to Random Fields
Authors:
Moo K. Chung
Abstract:
General linear models (GLM) are often constructed and used in statistical inference at the voxel level in brain imaging. In this paper, we explore the basics of random fields and the multiple comparisons on the random fields, which are necessary to properly threshold statistical maps for the whole image at specific statistical significance level. The multiple comparisons are crucial in determining…
▽ More
General linear models (GLM) are often constructed and used in statistical inference at the voxel level in brain imaging. In this paper, we explore the basics of random fields and the multiple comparisons on the random fields, which are necessary to properly threshold statistical maps for the whole image at specific statistical significance level. The multiple comparisons are crucial in determining overall statistical significance in correlated test statistics over the whole brain. In practice, t- or F-statistics in adjacent voxels are correlated. So there is the problem of multiple comparisons, which we have simply neglected up to now. For multiple comparisons that account for spatially correlated test statistics, various methods were proposed: Bonferroni correction, random field theory, false discovery rates and permutation tests. Among them, we will explore the random field approach.
△ Less
Submitted 19 July, 2020;
originally announced July 2020.
-
Sampled Limited Memory Methods for Massive Linear Inverse Problems
Authors:
Julianne Chung,
Matthias Chung,
J. Tanner Slagel,
Luis Tenorio
Abstract:
In many modern imaging applications the desire to reconstruct high resolution images, coupled with the abundance of data from acquisition using ultra-fast detectors, have led to new challenges in image reconstruction. A main challenge is that the resulting linear inverse problems are massive. The size of the forward model matrix exceeds the storage capabilities of computer memory, or the observati…
▽ More
In many modern imaging applications the desire to reconstruct high resolution images, coupled with the abundance of data from acquisition using ultra-fast detectors, have led to new challenges in image reconstruction. A main challenge is that the resulting linear inverse problems are massive. The size of the forward model matrix exceeds the storage capabilities of computer memory, or the observational dataset is enormous and not available all at once. Row-action methods that iterate over samples of rows can be used to approximate the solution while avoiding memory and data availability constraints. However, their overall convergence can be slow. In this paper, we introduce a sampled limited memory row-action method for linear least squares problems, where an approximation of the global curvature of the underlying least squares problem is used to speed up the initial convergence and to improve the accuracy of iterates. We show that this limited memory method is a generalization of the damped block Kaczmarz method, and we prove linear convergence of the expectation of the iterates and of the error norm up to a convergence horizon. Numerical experiments demonstrate the benefits of these sampled limited memory row-action methods for massive 2D and 3D inverse problems in tomography applications.
△ Less
Submitted 17 December, 2019;
originally announced December 2019.
-
Sampled Tikhonov Regularization for Large Linear Inverse Problems
Authors:
J. Tanner Slagel,
Julianne Chung,
Matthias Chung,
David Kozak,
Luis Tenorio
Abstract:
In this paper, we investigate iterative methods that are based on sampling of the data for computing Tikhonov-regularized solutions. We focus on very large inverse problems where access to the entire data set is not possible all at once (e.g., for problems with streaming or massive datasets). Row-access methods provide an ideal framework for solving such problems, since they only require access to…
▽ More
In this paper, we investigate iterative methods that are based on sampling of the data for computing Tikhonov-regularized solutions. We focus on very large inverse problems where access to the entire data set is not possible all at once (e.g., for problems with streaming or massive datasets). Row-access methods provide an ideal framework for solving such problems, since they only require access to "blocks" of the data at any given time. However, when using these iterative sampling methods to solve inverse problems, the main challenges include a proper choice of the regularization parameter, appropriate sampling strategies, and a convergence analysis. To address these challenges, we first describe a family of sampled iterative methods that can incorporate data as they become available (e.g., randomly sampled). We consider two sampled iterative methods, where the iterates can be characterized as solutions to a sequence of approximate Tikhonov problems. The first method requires the regularization parameter to be fixed a priori and converges asymptotically to an unregularized solution for randomly sampled data. This is undesirable for inverse problems. Thus, we focus on the second method where the main benefits are that the regularization parameter can be updated during the iterative process and the iterates converge asymptotically to a Tikhonov-regularized solution. We describe adaptive approaches to update the regularization parameter that are based on sampled residuals, and we describe a limited-memory variant for larger problems. Numerical examples, including a large-scale super-resolution imaging example, demonstrate the potential for these methods.
△ Less
Submitted 14 December, 2018;
originally announced December 2018.
-
Hyper $b$-ary expansions and Stern polynomials
Authors:
Tanay Wakhare,
Caleb Kendrick,
Matthew Chung,
Catherine Cassell,
Stefano Santini,
William Colin Mosley,
Anand Raghu,
Robert Morrison,
Iman Schurman,
Timothy Kevin Beal,
Matthew Patrick
Abstract:
We study a recently introduced base $b$ polynomial analog of Stern's diatomic sequence, which generalizes Stern polynomials of Klavar, Dilcher, Ericksen, Mansour, Stolarsky, and others. We lift some basic properties of base $2$ Stern polynomials to arbitrary base, and introduce a matrix characterization of Stern polynomials. By specializing, we recover some new number theoretic results about hyper…
▽ More
We study a recently introduced base $b$ polynomial analog of Stern's diatomic sequence, which generalizes Stern polynomials of Klavar, Dilcher, Ericksen, Mansour, Stolarsky, and others. We lift some basic properties of base $2$ Stern polynomials to arbitrary base, and introduce a matrix characterization of Stern polynomials. By specializing, we recover some new number theoretic results about hyper $b$-ary partitions, which count partitions of $n$ into powers of $b$.
△ Less
Submitted 25 October, 2018;
originally announced October 2018.
-
Parameter and Uncertainty Estimation for Dynamical Systems Using Surrogate Stochastic Processes
Authors:
M. Chung,
M. Binois,
R. B. Gramacy,
D. J. Moquin,
A. P. Smith,
A. M. Smith
Abstract:
Inference on unknown quantities in dynamical systems via observational data is essential for providing meaningful insight, furnishing accurate predictions, enabling robust control, and establishing appropriate designs for future experiments. Merging mathematical theory with empirical measurements in a statistically coherent way is critical and challenges abound, e.g.,: ill-posedness of the paramet…
▽ More
Inference on unknown quantities in dynamical systems via observational data is essential for providing meaningful insight, furnishing accurate predictions, enabling robust control, and establishing appropriate designs for future experiments. Merging mathematical theory with empirical measurements in a statistically coherent way is critical and challenges abound, e.g.,: ill-posedness of the parameter estimation problem, proper regularization and incorporation of prior knowledge, and computational limitations on full uncertainty qualification. To address these issues, we propose a new method for learning parameterized dynamical systems from data. In many ways, our proposal turns the canonical framework on its head. We first fit a surrogate stochastic process to observational data, enforcing prior knowledge (e.g., smoothness), and coping with challenging data features like heteroskedasticity, heavy tails and censoring. Then, samples of the stochastic process are used as "surrogate data" and point estimates are computed via ordinary point estimation methods in a modular fashion. An attractive feature of this approach is that it is fully Bayesian and simultaneously parallelizable. We demonstrate the advantages of our new approach on a predator prey simulation study and on a real world application involving within-host influenza virus infection data paired with a viral kinetic model.
△ Less
Submitted 2 February, 2018;
originally announced February 2018.
-
Optimal Experimental Design for Constrained Inverse Problems
Authors:
Lars Ruthotto,
Julianne Chung,
Matthias Chung
Abstract:
In this paper, we address the challenging problem of optimal experimental design (OED) of constrained inverse problems. We consider two OED formulations that allow reducing the experimental costs by minimizing the number of measurements. The first formulation assumes a fine discretization of the design parameter space and uses sparsity promoting regularization to obtain an efficient design. The se…
▽ More
In this paper, we address the challenging problem of optimal experimental design (OED) of constrained inverse problems. We consider two OED formulations that allow reducing the experimental costs by minimizing the number of measurements. The first formulation assumes a fine discretization of the design parameter space and uses sparsity promoting regularization to obtain an efficient design. The second formulation parameterizes the design and seeks optimal placement for these measurements by solving a small-dimensional optimization problem. We consider both problems in a Bayes risk as well as an empirical Bayes risk minimization framework. For the unconstrained inverse state problem, we exploit the closed form solution for the inner problem to efficiently compute derivatives for the outer OED problem. The empirical formulation does not require an explicit solution of the inverse problem and therefore allows to integrate constraints efficiently. A key contribution is an efficient optimization method for solving the resulting, typically high-dimensional, bilevel optimization problem using derivative-based methods. To overcome the lack of non-differentiability in active set methods for inequality constraints problems, we use a relaxed interior point method. To address the growing computational complexity of empirical Bayes OED, we parallelize the computation over the training models. Numerical examples and illustrations from tomographic reconstruction, for various data sets and under different constraints, demonstrate the impact of constraints on the optimal design and highlight the importance of OED for constrained problems.
△ Less
Submitted 15 August, 2017;
originally announced August 2017.
-
Large Deviation Principle for $S$-unimodal maps with flat critical point
Authors:
Yong Moo Chung,
Hiroki Takahasi
Abstract:
We study a topologically exact, negative Schwarzian unimodal map whose critical point is non-recurrent and flat. Assuming the critical order is either logarithmic or polynomial, we establish the Large Deviation Principle and give a partial description of the zeros of the corresponding rate functions. We apply our main results to a certain parametrized family of unimodal maps in the same topologica…
▽ More
We study a topologically exact, negative Schwarzian unimodal map whose critical point is non-recurrent and flat. Assuming the critical order is either logarithmic or polynomial, we establish the Large Deviation Principle and give a partial description of the zeros of the corresponding rate functions. We apply our main results to a certain parametrized family of unimodal maps in the same topological conjugacy class, and give a complete description of the zeros of the rate functions. We observe a qualitative change at a transition parameter, and show that the sets of zeros depend continuously on the parameter even at the transition.
△ Less
Submitted 17 December, 2017; v1 submitted 11 August, 2017;
originally announced August 2017.
-
Stochastic Newton and Quasi-Newton Methods for Large Linear Least-squares Problems
Authors:
Julianne Chung,
Matthias Chung,
J. Tanner Slagel,
Luis Tenorio
Abstract:
We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity is introduced in two different frameworks as a means to overcome these computat…
▽ More
We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity is introduced in two different frameworks as a means to overcome these computational limitations, and probability distributions that can exploit structure and/or sparsity are considered. Theoretical results on consistency of the approximations for both the stochastic Newton and the stochastic quasi-Newton methods are provided. The results show, in particular, that stochastic Newton iterates, in contrast to stochastic quasi-Newton iterates, may not converge to the desired least-squares solution. Numerical examples, including an example from extreme learning machines, demonstrate the potential applications of these methods.
△ Less
Submitted 23 February, 2017;
originally announced February 2017.
-
Large deviation principle in one-dimensional dynamics
Authors:
Yong Moo Chung,
Juan Rivera-Letelier,
Hiroki Takahasi
Abstract:
We study the dynamics of smooth interval maps with non-flat critical points. For every such a map that is topologically exact, we establish the full (level-2) Large Deviation Principle for empirical means. In particular, the Large Deviation Principle holds for every non\nobreakdash-renormalizable quadratic map. This includes the maps without physical measure found by Hofbauer and Keller, and chall…
▽ More
We study the dynamics of smooth interval maps with non-flat critical points. For every such a map that is topologically exact, we establish the full (level-2) Large Deviation Principle for empirical means. In particular, the Large Deviation Principle holds for every non\nobreakdash-renormalizable quadratic map. This includes the maps without physical measure found by Hofbauer and Keller, and challenges the widely-shared view of the Large Deviation Principle as a refinement of laws of large numbers.
△ Less
Submitted 17 July, 2019; v1 submitted 3 October, 2016;
originally announced October 2016.
-
Optimal regularized inverse matrices for inverse problems
Authors:
Julianne Chung,
Matthias Chung
Abstract:
In this paper, we consider optimal low-rank regularized inverse matrix approximations and their applications to inverse problems. We give an explicit solution to a generalized rank-constrained regularized inverse approximation problem, where the key novelties are that we allow for updates to existing approximations and we can incorporate additional probability distribution information. Since compu…
▽ More
In this paper, we consider optimal low-rank regularized inverse matrix approximations and their applications to inverse problems. We give an explicit solution to a generalized rank-constrained regularized inverse approximation problem, where the key novelties are that we allow for updates to existing approximations and we can incorporate additional probability distribution information. Since computing optimal regularized inverse matrices under rank constraints can be challenging, especially for problems where matrices are large and sparse or are only accessable via function call, we propose an efficient rank-update approach that decomposes the problem into a sequence of smaller rank problems. Using examples from image deblurring, we demonstrate that more accurate solutions to inverse problems can be achieved by using rank-updates to existing regularized inverse approximations. Furthermore, we show the potential benefits of using optimal regularized inverse matrix updates for solving perturbed tomographic reconstruction problems.
△ Less
Submitted 18 March, 2016;
originally announced March 2016.
-
Robust Parameter Estimation for Biological Systems: A Study on the Dynamics of Microbial Communities
Authors:
Matthias Chung,
Justin Krueger,
Mihai Pop
Abstract:
Interest in the study of in-host microbial communities has increased in recent years due to our improved understanding of the communities' significant role in host health. As a result, the ability to model these communities using differential equations, for example, and analyze the results has become increasingly relevant. The size of the models and limitations in data collection among many other…
▽ More
Interest in the study of in-host microbial communities has increased in recent years due to our improved understanding of the communities' significant role in host health. As a result, the ability to model these communities using differential equations, for example, and analyze the results has become increasingly relevant. The size of the models and limitations in data collection among many other considerations require that we develop new parameter estimation methods to address the challenges that arise when using traditional parameter estimation methods for models of these in-host microbial communities. In this work, we present the challenges that appear when applying traditional parameter estimation techniques to differential equation models of microbial communities, and we provide an original, alternative method to those techniques. We show the derivation of our method and how our method avoids the limitations of traditional techniques while including additional benefits. We also provide simulation studies to demonstrate our method's viability, the application of our method to a model of intestinal microbial communities to demonstrate the insights that can be gained from our method, and sample code to give readers the opportunity to apply our method to their own research.
△ Less
Submitted 23 September, 2015;
originally announced September 2015.
-
An Efficient Approach for Computing Optimal Low-Rank Regularized Inverse Matrices
Authors:
Julianne Chung,
Matthias Chung
Abstract:
Standard regularization methods that are used to compute solutions to ill-posed inverse problems require knowledge of the forward model. In many real-life applications, the forward model is not known, but training data is readily available. In this paper, we develop a new framework that uses training data, as a substitute for knowledge of the forward model, to compute an optimal low-rank regulariz…
▽ More
Standard regularization methods that are used to compute solutions to ill-posed inverse problems require knowledge of the forward model. In many real-life applications, the forward model is not known, but training data is readily available. In this paper, we develop a new framework that uses training data, as a substitute for knowledge of the forward model, to compute an optimal low-rank regularized inverse matrix directly, allowing for very fast computation of a regularized solution. We consider a statistical framework based on Bayes and empirical Bayes risk minimization to analyze theoretical properties of the problem. We propose an efficient rank update approach for computing an optimal low-rank regularized inverse matrix for various error measures. Numerical experiments demonstrate the benefits and potential applications of our approach to problems in signal and image processing.
△ Less
Submitted 6 April, 2014;
originally announced April 2014.
-
Multifractal formalism for Benedicks-Carleson quadratic maps
Authors:
Yong Moo Chung,
Hiroki Takahasi
Abstract:
For a positive measure set of nonuniformly expanding quadratic maps on the interval we effect a multifractal formalism, i.e., decompose the phase space into level sets of time averages of a given observable and consider the associated {\it Birkhoff spectrum} which encodes this decomposition. We derive a formula which relates the Hausdorff dimension of level sets to entropies and Lyapunov exponents…
▽ More
For a positive measure set of nonuniformly expanding quadratic maps on the interval we effect a multifractal formalism, i.e., decompose the phase space into level sets of time averages of a given observable and consider the associated {\it Birkhoff spectrum} which encodes this decomposition. We derive a formula which relates the Hausdorff dimension of level sets to entropies and Lyapunov exponents of invariant probability measures, and then use this formula to show that the spectrum is continuous. In order to estimate the Hausdorff dimension from above, one has to "see" sufficiently many points. To this end, we construct a family of towers. Using these towers we establish a large deviation principle for empirical distributions, with Lebesgue as a reference measure.
△ Less
Submitted 13 November, 2012; v1 submitted 8 December, 2011;
originally announced December 2011.
-
Large deviation principle for Benedicks-Carleson quadratic maps
Authors:
Yong Moo Chung,
Hiroki Takahasi
Abstract:
Since the pioneering works of Jakobson and Benedicks & Carleson and others, it has been known that a positive measure set of quadratic maps admit invariant probability measures absolutely continuous with respect to Lebesgue. These measures allow one to statistically predict the asymptotic fate of Lebesgue almost every initial condition. Estimating fluctuations of empirical distributions before the…
▽ More
Since the pioneering works of Jakobson and Benedicks & Carleson and others, it has been known that a positive measure set of quadratic maps admit invariant probability measures absolutely continuous with respect to Lebesgue. These measures allow one to statistically predict the asymptotic fate of Lebesgue almost every initial condition. Estimating fluctuations of empirical distributions before they settle to equilibrium requires a fairly good control over large parts of the phase space. We use the sub-exponential slow recurrence condition of Benedicks & Carleson to build induced Markov maps of arbitrarily small scale and associated towers, to which the absolutely continuous measures can be lifted. These various lifts together enable us to obtain a control of recurrence that is sufficient to establish a level 2 large deviation principle, for the absolutely continuous measures. This result encompasses dynamics far from equilibrium, and thus significantly extends presently known local large deviations results for quadratic maps.
△ Less
Submitted 12 September, 2012; v1 submitted 22 June, 2011;
originally announced June 2011.
-
Gamma-based clustering via ordered means with application to gene-expression analysis
Authors:
Michael A. Newton,
Lisa M. Chung
Abstract:
Discrete mixture models provide a well-known basis for effective clustering algorithms, although technical challenges have limited their scope. In the context of gene-expression data analysis, a model is presented that mixes over a finite catalog of structures, each one representing equality and inequality constraints among latent expected values. Computations depend on the probability that indepe…
▽ More
Discrete mixture models provide a well-known basis for effective clustering algorithms, although technical challenges have limited their scope. In the context of gene-expression data analysis, a model is presented that mixes over a finite catalog of structures, each one representing equality and inequality constraints among latent expected values. Computations depend on the probability that independent gamma-distributed variables attain each of their possible orderings. Each ordering event is equivalent to an event in independent negative-binomial random variables, and this finding guides a dynamic-programming calculation. The structuring of mixture-model components according to constraints among latent means leads to strict concavity of the mixture log likelihood. In addition to its beneficial numerical properties, the clustering method shows promising results in an empirical study.
△ Less
Submitted 9 November, 2012; v1 submitted 22 July, 2009;
originally announced July 2009.
-
Birkhoff spectra for one-dimensional maps with some hyperbolicity
Authors:
Yong Moo Chung
Abstract:
We study the multifractal analysis for smooth dynamical systems in dimension one. It is characterized the Hausdorff dimension of the level set obtained from the Birkhoff averages of a continuous function by the local dimensions of hyperbolic measures for a topologically mixing $C^2$ map modelled by an abstract dynamical system. A characterization which corresponds to above is also given for the…
▽ More
We study the multifractal analysis for smooth dynamical systems in dimension one. It is characterized the Hausdorff dimension of the level set obtained from the Birkhoff averages of a continuous function by the local dimensions of hyperbolic measures for a topologically mixing $C^2$ map modelled by an abstract dynamical system. A characterization which corresponds to above is also given for the ergodic basins of invariant probability measures. And it is shown that the complement of the set of quasi-regular points carries full Hausdorff dimension.
△ Less
Submitted 11 March, 2008;
originally announced March 2008.
-
Recurrence times and large deviations
Authors:
Yong Moo Chung
Abstract:
We give a criterion to determine the large deviation rate functions for abstract dynamical systems on towers. As an application of this criterion we show the level 2 large deviation principle for some class of smooth interval maps with nonuniform hyperbolicity.
We give a criterion to determine the large deviation rate functions for abstract dynamical systems on towers. As an application of this criterion we show the level 2 large deviation principle for some class of smooth interval maps with nonuniform hyperbolicity.
△ Less
Submitted 15 January, 2008;
originally announced January 2008.
