-
Assessing the performance of spatial cross-validation approaches for models of spatially structured data
Authors:
Michael J Mahoney,
Lucas K Johnson,
Julia Silge,
Hannah Frick,
Max Kuhn,
Colin M Beier
Abstract:
Evaluating models fit to data with internal spatial structure requires specific cross-validation (CV) approaches, because randomly selecting assessment data may produce assessment sets that are not truly independent of data used to train the model. Many spatial CV methodologies have been proposed to address this by forcing models to extrapolate spatially when predicting the assessment set. However…
▽ More
Evaluating models fit to data with internal spatial structure requires specific cross-validation (CV) approaches, because randomly selecting assessment data may produce assessment sets that are not truly independent of data used to train the model. Many spatial CV methodologies have been proposed to address this by forcing models to extrapolate spatially when predicting the assessment set. However, to date there exists little guidance on which methods yield the most accurate estimates of model performance.
We conducted simulations to compare model performance estimates produced by five common CV methods fit to spatially structured data. We found spatial CV approaches generally improved upon resubstitution and V-fold CV estimates, particularly when approaches which combined assessment sets of spatially conjunct observations with spatial exclusion buffers. To facilitate use of these techniques, we introduce the `spatialsample` package which provides tooling for performing spatial CV as part of the broader tidymodels modeling framework.
△ Less
Submitted 13 March, 2023;
originally announced March 2023.
-
On the maximum likelihood estimation in general log-linear models
Authors:
Anna Klimova,
Matthias Kuhn
Abstract:
General log-linear models specified by non-negative integer design matrices have a potentially wide range of applications, although using models without the genuine overall effect, that is, ones which cannot be reparameterized to include a normalizing constant, is still rare. The log-linear models without the overall effect arise naturally in practice, and can be handled in a similar manner to mod…
▽ More
General log-linear models specified by non-negative integer design matrices have a potentially wide range of applications, although using models without the genuine overall effect, that is, ones which cannot be reparameterized to include a normalizing constant, is still rare. The log-linear models without the overall effect arise naturally in practice, and can be handled in a similar manner to models with the overall effect. A novel iterative scaling procedure for the MLE computation under such models is proposed, and its convergence is proved. The results are illustrated using data from a recent clinical study.
△ Less
Submitted 30 December, 2022; v1 submitted 25 August, 2022;
originally announced August 2022.
-
A Non-ergodic Effective Amplitude Ground-Motion Model for California
Authors:
Grigorios Lavrentiadis,
Norman A. Abrahamson,
Nicolas M. Kuehn
Abstract:
A new non-ergodic ground-motion model (GMM) for effective amplitude spectral ($EAS$) values for California is presented in this study. $EAS$, which is defined in Goulet et al. (2018), is a smoothed rotation-independent Fourier amplitude spectrum of the two horizontal components of an acceleration time history. The main motivation for developing a non-ergodic $EAS$ GMM, rather than a spectral accel…
▽ More
A new non-ergodic ground-motion model (GMM) for effective amplitude spectral ($EAS$) values for California is presented in this study. $EAS$, which is defined in Goulet et al. (2018), is a smoothed rotation-independent Fourier amplitude spectrum of the two horizontal components of an acceleration time history. The main motivation for developing a non-ergodic $EAS$ GMM, rather than a spectral acceleration GMM, is that the scaling of $EAS$ does not depend on spectral shape, and therefore, the more frequent small magnitude events can be used in the estimation of the non-ergodic terms.
The model is developed using the California subset of the NGAWest2 dataset Ancheta et al. (2013). The Bayless and Abrahamson (2019b) (BA18) ergodic $EAS$ GMM was used as backbone to constrain the average source, path, and site scaling. The non-ergodic GMM is formulated as a Bayesian hierarchical model: the non-ergodic source and site terms are modeled as spatially varying coefficients following the approach of Landwehr et al. (2016), and the non-ergodic path effects are captured by the cell-specific anelastic attenuation attenuation following the approach of Dawood and Rodriguez-Marek (2013). Close to stations and past events, the mean values of the non-ergodic terms deviate from zero to capture the systematic effects and their epistemic uncertainty is small. In areas with sparse data, the epistemic uncertainty of the non-ergodic terms is large, as the systematic effects cannot be determined.
The non-ergodic total aleatory standard deviation is approximately $30$ to $40\%$ smaller than the total aleatory standard deviation of BA18. This reduction in the aleatory variability has a significant impact on hazard calculations at large return periods. The epistemic uncertainty of the ground motion predictions is small in areas close to stations and past events.
△ Less
Submitted 23 June, 2021; v1 submitted 14 June, 2021;
originally announced June 2021.
-
Best Practices for Alchemical Free Energy Calculations
Authors:
Antonia S. J. S. Mey,
Bryce Allen,
Hannah E. Bruce Macdonald,
John D. Chodera,
Maximilian Kuhn,
Julien Michel,
David L. Mobley,
Levi N. Naden,
Samarjeet Prasad,
Andrea Rizzi,
Jenke Scheen,
Michael R. Shirts,
Gary Tresadern,
Huafeng Xu
Abstract:
Alchemical free energy calculations are a useful tool for predicting free energy differences associated with the transfer of molecules from one environment to another. The hallmark of these methods is the use of "bridging" potential energy functions representing \emph{alchemical} intermediate states that cannot exist as real chemical species. The data collected from these bridging alchemical therm…
▽ More
Alchemical free energy calculations are a useful tool for predicting free energy differences associated with the transfer of molecules from one environment to another. The hallmark of these methods is the use of "bridging" potential energy functions representing \emph{alchemical} intermediate states that cannot exist as real chemical species. The data collected from these bridging alchemical thermodynamic states allows the efficient computation of transfer free energies (or differences in transfer free energies) with orders of magnitude less simulation time than simulating the transfer process directly. While these methods are highly flexible, care must be taken in avoiding common pitfalls to ensure that computed free energy differences can be robust and reproducible for the chosen force field, and that appropriate corrections are included to permit direct comparison with experimental data. In this paper, we review current best practices for several popular application domains of alchemical free energy calculations, including relative and absolute small molecule binding free energy calculations to biomolecular targets.
△ Less
Submitted 21 August, 2020; v1 submitted 7 August, 2020;
originally announced August 2020.
-
Inter-Node Distance Estimation from Multipath Delay Differences of Channels to Observer Nodes
Authors:
Gregor Dumphart,
Marc Kuhn,
Armin Wittneben,
Florian Trösch
Abstract:
We study the estimation of distance d between two wireless nodes by means of their wideband channels to a third node, called observer. The motivating principle is that the channel impulse responses are similar for small d and drift apart when d increases. Following this idea we propose specific distance estimators based on the differences of path delays of the extractable multipath components. In…
▽ More
We study the estimation of distance d between two wireless nodes by means of their wideband channels to a third node, called observer. The motivating principle is that the channel impulse responses are similar for small d and drift apart when d increases. Following this idea we propose specific distance estimators based on the differences of path delays of the extractable multipath components. In particular, we derive such estimators for rich multipath environments and various important cases: with and without clock synchronization as well as errors on the extracted path delays (e.g. due to limited bandwidth). The estimators readily support (and benefit from) the presence of multiple observers. We present an error analysis and, using ray tracing in an exemplary indoor environment, show that the estimators perform well in realistic conditions. We describe possible localization applications of the proposed scheme and highlight its major advantages: it requires neither precise synchronization nor line-of-sight connection. This could make wireless user tracking feasible in dynamic indoor settings.
△ Less
Submitted 12 February, 2019;
originally announced February 2019.
-
Mixture Models in Astronomy
Authors:
Michael A. Kuhn,
Eric D. Feigelson
Abstract:
Mixture models combine multiple components into a single probability density function. They are a natural statistical model for many situations in astronomy, such as surveys containing multiple types of objects, cluster analysis in various data spaces, and complicated distribution functions. This chapter in the CRC Handbook of Mixture Analysis is concerned with astronomical applications of mixture…
▽ More
Mixture models combine multiple components into a single probability density function. They are a natural statistical model for many situations in astronomy, such as surveys containing multiple types of objects, cluster analysis in various data spaces, and complicated distribution functions. This chapter in the CRC Handbook of Mixture Analysis is concerned with astronomical applications of mixture models for cluster analysis, classification, and semi-parametric density estimation. We present several classification examples from the literature, including identification of a new class, analysis of contaminants, and overlapping populations. In most cases, mixtures of normal (Gaussian) distributions are used, but it is sometimes necessary to use different distribution functions derived from astrophysical experience. We also address the use of mixture models for the analysis of spatial distributions of objects, like galaxies in redshift surveys or young stars in star-forming regions. In the case of galaxy clustering, mixture models may not be the optimal choice for understanding the homogeneous and isotropic structure of voids and filaments. However, we show that mixture models, using astrophysical models for star clusters, may provide a natural solution to the problem of subdividing a young stellar population into subclusters. Finally, we explore how mixture models can be used for mathematically advanced modeling of data with heteroscedastic uncertainties or missing values, providing two example algorithms, the measurement error regression model of Kelly (2007) and the Extreme Deconvolution model of Bovy et al. (2011). The challenges presented by astronomical science, aided by the public availability of catalogs from major surveys and missions, are a rich area for collaboration between statisticians and astronomers.
△ Less
Submitted 28 January, 2019; v1 submitted 29 November, 2017;
originally announced November 2017.
-
Futility Analysis in the Cross-Validation of Machine Learning Models
Authors:
Max Kuhn
Abstract:
Many machine learning models have important structural tuning parameters that cannot be directly estimated from the data. The common tactic for setting these parameters is to use resampling methods, such as cross--validation or the bootstrap, to evaluate a candidate set of values and choose the best based on some pre--defined criterion. Unfortunately, this process can be time consuming. However, t…
▽ More
Many machine learning models have important structural tuning parameters that cannot be directly estimated from the data. The common tactic for setting these parameters is to use resampling methods, such as cross--validation or the bootstrap, to evaluate a candidate set of values and choose the best based on some pre--defined criterion. Unfortunately, this process can be time consuming. However, the model tuning process can be streamlined by adaptively resampling candidate values so that settings that are clearly sub-optimal can be discarded. The notion of futility analysis is introduced in this context. An example is shown that illustrates how adaptive resampling can be used to reduce training time. Simulation studies are used to understand how the potential speed--up is affected by parallel processing techniques.
△ Less
Submitted 27 May, 2014;
originally announced May 2014.
