Even with the rise in popularity of over-parameterized models, simple dimensionality reduction an... more Even with the rise in popularity of over-parameterized models, simple dimensionality reduction and clustering methods, such as PCA and k-means, are still routinely used in an amazing variety of settings. A primary reason is the combination of simplicity, interpretability and computational efficiency. The focus of this article is on improving upon PCA and k-means, by allowing non-linear relations in the data and more flexible cluster shapes, without sacrificing the key advantages. The key contribution is a new framework for Principal Elliptical Analysis (PEA), defining a simple and computationally efficient alternative to PCA that fits the best elliptical approximation through the data. We provide theoretical guarantees on the proposed PEA algorithm using Vapnik-Chervonenkis (VC) theory to show strong consistency and uniform concentration bounds. Toy experiments illustrate the performance of PEA, and the ability to adapt to non-linear structure and complex cluster shapes. In a rich v...
Starting with a carefully formulated Dirichlet process (DP) mixture model, we derive a generalize... more Starting with a carefully formulated Dirichlet process (DP) mixture model, we derive a generalized product partition model (GPPM) in which the parti- tion process is predictor-dependent. The GPPM generalizes DP clustering to relax the exchangeability assumption through the incorporation of predictors, resulting in a generalized Polya urn scheme. In addition, the GPPM can be used for formu- lating flexible semiparametric Bayes models for conditional distribution estimation, bypassing the need for expensive computation of large numbers of unknowns charac- terizing priors for dependent collections of random probability measures. A variety of special cases are considered, and an efficient Gibbs sampling algorithm is de- veloped for posterior computation. The methods are illustrated using simulation examples and an epidemiologic application.
In many modern applications, there is interest in analyzing enormous data sets that cannot be eas... more In many modern applications, there is interest in analyzing enormous data sets that cannot be easily moved across computers or loaded into memory on a single computer. In such settings, it is very common to be interested in clustering. Existing distributed clustering algorithms are mostly distance or density based without a likelihood specification, precluding the possibility of formal statistical inference. We introduce a nearly embarrassingly parallel algorithm using a Bayesian finite mixture of mixtures model for distributed clustering, which we term distributed Bayesian clustering (DIB-C). DIB-C can flexibly accommodate data sets with various shapes (e.g. skewed or multi-modal). With data randomly partitioned and distributed, we first run Markov chain Monte Carlo in an embarrassingly parallel manner to obtain local clustering draws and then refine across nodes for a final cluster estimate based on any loss function on the space of partitions. DIB-C can also provide a posterior p...
Closed surfaces provide a useful model for 3-d shapes, with the data typically consisting of a cl... more Closed surfaces provide a useful model for 3-d shapes, with the data typically consisting of a cloud of points in R. The existing literature on closed surface modeling focuses on frequentist point estimation methods that join surface patches along the edges, with surface patches created via Bézier surfaces or tensor products of B-splines. However, the resulting surfaces are not smooth along the edges and the geometric constraints required to join the surface patches lead to computational drawbacks. In this article, we develop a Bayesian model for closed surfaces based on tensor products of a cyclic basis resulting in infinitely smooth surface realizations. We impose sparsity on the control points through a doubleshrinkage prior. Theoretical properties of the support of our proposed prior are studied and it is shown that the posterior achieves the optimal rate of convergence under reasonable assumptions on the prior. The proposed approach is illustrated with some examples.
We consider the problem for which K different types of data are collected to characterize an asso... more We consider the problem for which K different types of data are collected to characterize an associated inference task, with this performed for M distinct tasks. It is assumed that the parameters associated with the model for data type (modality) k may be represented in the form of a mixture model, with the M tasks representing M draws from the mixture. We wish to simultaneously infer mixture models across all K modality types, using data from all M tasks. Considering tasks m1 and m2, we wish to impose the belief that if the data associated with modality k are drawn from the same mixture component (implying a similarity between tasks m1 and m2), then it is more probable that the associated data from modality j 6= k will also be drawn from the same component. On the other hand, it is anticipated that there may be “random effects” that manifest idiosyncratic behavior for a subset of the modalities, even when similarity exists between the other modalities. The model employed utilizes a...
For massive data sets, efficient computation commonly relies on distributed algorithms that store... more For massive data sets, efficient computation commonly relies on distributed algorithms that store and process subsets of the data on different machines, minimizing communication costs. Our focus is on regression and classification problems involving many features. A variety of distributed algorithms have been proposed in this context, but challenges arise in defining an algorithm with low communication, theoretical guarantees and excellent practical performance in general settings. We propose a MEdian Selection Subset AGgregation Estimator (message) algorithm, which attempts to solve these problems. The algorithm applies feature selection in parallel for each subset using Lasso or another method, calculates the 'median' feature inclusion index, estimates coefficients for the selected features in parallel for each subset, and then averages these estimates. The algorithm is simple, involves very minimal communication, scales efficiently in both sample and feature size, and has...
We consider problems for which one has incomplete binary matrices that evolve with time (e.g., th... more We consider problems for which one has incomplete binary matrices that evolve with time (e.g., the votes of legislators on particular legislation, with each year characterized by a different such matrix). An objective of such analysis is to infer structure and inter-relationships underlying the matrices, here defined by latent features associated with each axis of the matrix. In addition, it is assumed that documents are available for the entities associated with at least one of the matrix axes. By jointly analyzing the matrices and documents, one may be used to inform the other within the analysis, and the model offers the opportunity to predict matrix values (e.g., votes) based only on an associated document (e.g., legislation). The research presented here merges two areas of machine-learning that have previously been investigated separately: incomplete-matrix analysis and topic modeling. The analysis is performed from a Bayesian perspective, with efficient inference constituted v...
We propose a general scheme for solving convex and non-convex optimization problems on manifolds.... more We propose a general scheme for solving convex and non-convex optimization problems on manifolds. The central idea is that, by adding a multiple of the squared retraction distance to the objective function in question, we "convexify" the objective function and solve a series of convex sub-problems in the optimization procedure. One of the key challenges for optimization on manifolds is the difficulty of verifying the complexity of the objective function, e.g., whether the objective function is convex or non-convex, and the degree of non-convexity. Our proposed algorithm adapts to the level of complexity in the objective function. We show that when the objective function is convex, the algorithm provably converges to the optimum and leads to accelerated convergence. When the objective function is non-convex, the algorithm will converge to a stationary point. Our proposed method unifies insights from Nesterov's original idea for accelerating gradient descent algorithms w...
Although there are many methods for functional data analysis (FDA), little emphasis is put on cha... more Although there are many methods for functional data analysis (FDA), little emphasis is put on characterizing variability among volatilities of individual functions. In particular, certain individuals exhibit erratic swings in their trajectory while other individuals have more stable trajectories. There is evidence of such volatility heterogeneity in blood pressure trajectories during pregnancy, for example, and reason to suspect that volatility is a biologically important feature. Most FDA models implicitly assume similar or identical smoothness of the individual functions, and hence can lead to misleading inferences on volatility and an inadequate representation of the functions. We propose a novel class of FDA models characterized using hierarchical stochastic differential equations. We model the derivatives of a mean function and deviation functions using Gaussian processes, while also allowing covariate dependence including on the volatilities of the deviation functions. Followi...
Models for distributions of shapes contained within images can be widely used in biomedical appli... more Models for distributions of shapes contained within images can be widely used in biomedical applications ranging from tumor tracking for targeted radiation therapy to classifying cells in a blood sample. Our focus is on hierarchical probability models for the shape and size of simply connected 2D closed curves, avoiding the need to specify landmarks through modeling the entire curve while borrowing information across curves for related objects. Prevalent approaches follow a fundamentally dierent strategy in providing an initial
In nonparametric regression, it is common for the inputs to fall in a restricted subset of Euclid... more In nonparametric regression, it is common for the inputs to fall in a restricted subset of Euclidean space. Typical kernel-based methods that do not take into account the intrinsic geometry of the domain across which observations are collected may produce sub-optimal results. In this article, we focus on solving this problem in the context of Gaussian process (GP) models, proposing a new class of Graph based GPs (GL-GPs), which learn a covariance that respects the geometry of the input domain. As the heat kernel is intractable computationally, we approximate the covariance using finitely-many eigenpairs of the Graph Laplacian (GL). The GL is constructed from a kernel which depends only on the Euclidean coordinates of the inputs. Hence, we can benefit from the full knowledge about the kernel to extend the covariance structure to newly arriving samples by a Nyström type extension. We provide substantial theoretical support for the GL-GP methodology, and illustrate performance gains in...
Markov chain Monte Carlo (MCMC) sampling algorithms have dominated the literature on posterior co... more Markov chain Monte Carlo (MCMC) sampling algorithms have dominated the literature on posterior computation. However, MCMC faces substantial hurdles in performing efficient posterior sampling for challenging Bayesian models, particularly in high-dimensional and large data settings. Motivated in part by such hurdles, an intriguing new class of piecewise deterministic Markov processes (PDMPs) has recently been proposed as an alternative to MCMC. One of the most popular types of PDMPs is known as the zig-zag (ZZ) sampler. Such algorithms require a computational upper bound in a Poisson thinning step, with performance improving for tighter bounds. In order to facilitate scaling to larger classes of problems, we propose a general class of Gibbs zig-zag (GZZ) samplers. GZZ allows parameters to be updated in blocks with ZZ applied to certain parameters and traditional MCMC style updates to others. This provides a flexible framework to combine PDMPs with the rich literature on MCMC algorithm...
We consider the problem of flexible modeling of higher order hidden Markov models when the number... more We consider the problem of flexible modeling of higher order hidden Markov models when the number of latent states and the nature of the serial dependence, including the true order, are unknown. We propose Bayesian nonparametric methodology based on tensor factorization techniques that can characterize any transition probability with a specified maximal order, allowing automated selection of the important lags and capturing higher order interactions among the lags. Theoretical results provide insights into identifiability of the emission distributions and asymptotic behavior of the posterior. We design efficient Markov chain Monte Carlo algorithms for posterior computation. In simulation experiments, the method vastly outperformed its first and higher order competitors not just in higher order settings, but, remarkably, also in first order cases. Practical utility is illustrated using real world applications. Some
The dimension of the parameter space is typically unknown in a variety of models that rely on fac... more The dimension of the parameter space is typically unknown in a variety of models that rely on factorizations. For example, in factor analysis the number of latent factors is not known and has to be inferred from the data. Although classical shrinkage priors are useful in such contexts, increasing shrinkage priors can provide a more effective approach that progressively penalizes expansions with growing complexity. In this article we propose a novel increasing shrinkage prior, called the cumulative shrinkage process, for the parameters that control the dimension in overcomplete formulations. Our construction has broad applicability and is based on an interpretable sequence of spike-and-slab distributions which assign increasing mass to the spike as the model complexity grows. Using factor analysis as an illustrative example, we show that this formulation has theoretical and practical advantages relative to current competitors, including an improved ability to recover the model dimens...
Fitting statistical models is computationally challenging when the sample size or the dimension o... more Fitting statistical models is computationally challenging when the sample size or the dimension of the dataset is huge. An attractive approach for down-scaling the problem size is to first partition the dataset into subsets and then fit using distributed algorithms. The dataset can be partitioned either horizontally (in the sample space) or vertically (in the feature space). While the majority of the literature focuses on sample space partitioning, feature space partitioning is more effective when $p\gg n$. Existing methods for partitioning features, however, are either vulnerable to high correlations or inefficient in reducing the model dimension. In this paper, we solve these problems through a new embarrassingly parallel framework named DECO for distributed variable selection and parameter estimation. In DECO, variables are first partitioned and allocated to $m$ distributed workers. The decorrelated subset data within each worker are then fitted via any algorithm designed for hig...
Many modern applications collect large sample size and highly imbalanced categorical data, with s... more Many modern applications collect large sample size and highly imbalanced categorical data, with some categories being relatively rare. Bayesian hierarchical models are well motivated in such settings in providing an approach to borrow information to combat data sparsity, while quantifying uncertainty in estimation. However, a fundamental problem is scaling up posterior computation to massive sample sizes. In categorical data models, posterior computation commonly relies on data augmentation Gibbs sampling. In this article, we study computational efficiency of such algorithms in a large sample imbalanced regime, showing that mixing is extremely poor, with a spectral gap that converges to zero at a rate proportional to the square root of sample size or faster. This theoretical result is verified with empirical performance in simulations and an application to a computational advertising data set. In contrast, algorithms that bypass data augmentation show rapid mixing on the same dataset.
In many applications, there is interest in clustering very high-dimensional data. A common strate... more In many applications, there is interest in clustering very high-dimensional data. A common strategy is first stage dimensionality reduction followed by a standard clustering algorithm, such as k-means. This approach does not target dimension reduction to the clustering objective, and fails to quantify uncertainty. Model-based Bayesian approaches provide an appealing alternative, but often have poor performance in high-dimensions, producing too many or too few clusters. This article provides an explanation for this behavior through studying the clustering posterior in a non-standard setting with fixed sample size and increasing dimensionality. We show that the finite sample posterior tends to either assign every observation to a different cluster or all observations to the same cluster as dimension grows, depending on the kernels and prior specification but not on the true data-generating model. To find models avoiding this pitfall, we define a Bayesian oracle for clustering, with th...
In multivariate data analysis, it is often important to estimate a graph characterizing dependenc... more In multivariate data analysis, it is often important to estimate a graph characterizing dependence among p variables. A popular strategy uses the non-zero entries in a p× p covariance or precision matrix, typically requiring restrictive modeling assumptions for accurate graph recovery. To improve model robustness, we instead focus on estimating the backbone of the dependence graph. We use a spanning tree likelihood, based on a minimalist graphical model that is purposely overly-simplified. Taking a Bayesian approach, we place a prior on the space of trees and quantify uncertainty in the graphical model. In both theory and experiments, we show that this model does not require the population graph to be a spanning tree or the covariance to satisfy assumptions beyond positive-definiteness. The model accurately recovers the backbone of the population graph at a rate competitive with existing approaches but with better robustness. We show combinatorial properties of the spanning tree, wh...
Even with the rise in popularity of over-parameterized models, simple dimensionality reduction an... more Even with the rise in popularity of over-parameterized models, simple dimensionality reduction and clustering methods, such as PCA and k-means, are still routinely used in an amazing variety of settings. A primary reason is the combination of simplicity, interpretability and computational efficiency. The focus of this article is on improving upon PCA and k-means, by allowing non-linear relations in the data and more flexible cluster shapes, without sacrificing the key advantages. The key contribution is a new framework for Principal Elliptical Analysis (PEA), defining a simple and computationally efficient alternative to PCA that fits the best elliptical approximation through the data. We provide theoretical guarantees on the proposed PEA algorithm using Vapnik-Chervonenkis (VC) theory to show strong consistency and uniform concentration bounds. Toy experiments illustrate the performance of PEA, and the ability to adapt to non-linear structure and complex cluster shapes. In a rich v...
Starting with a carefully formulated Dirichlet process (DP) mixture model, we derive a generalize... more Starting with a carefully formulated Dirichlet process (DP) mixture model, we derive a generalized product partition model (GPPM) in which the parti- tion process is predictor-dependent. The GPPM generalizes DP clustering to relax the exchangeability assumption through the incorporation of predictors, resulting in a generalized Polya urn scheme. In addition, the GPPM can be used for formu- lating flexible semiparametric Bayes models for conditional distribution estimation, bypassing the need for expensive computation of large numbers of unknowns charac- terizing priors for dependent collections of random probability measures. A variety of special cases are considered, and an efficient Gibbs sampling algorithm is de- veloped for posterior computation. The methods are illustrated using simulation examples and an epidemiologic application.
In many modern applications, there is interest in analyzing enormous data sets that cannot be eas... more In many modern applications, there is interest in analyzing enormous data sets that cannot be easily moved across computers or loaded into memory on a single computer. In such settings, it is very common to be interested in clustering. Existing distributed clustering algorithms are mostly distance or density based without a likelihood specification, precluding the possibility of formal statistical inference. We introduce a nearly embarrassingly parallel algorithm using a Bayesian finite mixture of mixtures model for distributed clustering, which we term distributed Bayesian clustering (DIB-C). DIB-C can flexibly accommodate data sets with various shapes (e.g. skewed or multi-modal). With data randomly partitioned and distributed, we first run Markov chain Monte Carlo in an embarrassingly parallel manner to obtain local clustering draws and then refine across nodes for a final cluster estimate based on any loss function on the space of partitions. DIB-C can also provide a posterior p...
Closed surfaces provide a useful model for 3-d shapes, with the data typically consisting of a cl... more Closed surfaces provide a useful model for 3-d shapes, with the data typically consisting of a cloud of points in R. The existing literature on closed surface modeling focuses on frequentist point estimation methods that join surface patches along the edges, with surface patches created via Bézier surfaces or tensor products of B-splines. However, the resulting surfaces are not smooth along the edges and the geometric constraints required to join the surface patches lead to computational drawbacks. In this article, we develop a Bayesian model for closed surfaces based on tensor products of a cyclic basis resulting in infinitely smooth surface realizations. We impose sparsity on the control points through a doubleshrinkage prior. Theoretical properties of the support of our proposed prior are studied and it is shown that the posterior achieves the optimal rate of convergence under reasonable assumptions on the prior. The proposed approach is illustrated with some examples.
We consider the problem for which K different types of data are collected to characterize an asso... more We consider the problem for which K different types of data are collected to characterize an associated inference task, with this performed for M distinct tasks. It is assumed that the parameters associated with the model for data type (modality) k may be represented in the form of a mixture model, with the M tasks representing M draws from the mixture. We wish to simultaneously infer mixture models across all K modality types, using data from all M tasks. Considering tasks m1 and m2, we wish to impose the belief that if the data associated with modality k are drawn from the same mixture component (implying a similarity between tasks m1 and m2), then it is more probable that the associated data from modality j 6= k will also be drawn from the same component. On the other hand, it is anticipated that there may be “random effects” that manifest idiosyncratic behavior for a subset of the modalities, even when similarity exists between the other modalities. The model employed utilizes a...
For massive data sets, efficient computation commonly relies on distributed algorithms that store... more For massive data sets, efficient computation commonly relies on distributed algorithms that store and process subsets of the data on different machines, minimizing communication costs. Our focus is on regression and classification problems involving many features. A variety of distributed algorithms have been proposed in this context, but challenges arise in defining an algorithm with low communication, theoretical guarantees and excellent practical performance in general settings. We propose a MEdian Selection Subset AGgregation Estimator (message) algorithm, which attempts to solve these problems. The algorithm applies feature selection in parallel for each subset using Lasso or another method, calculates the 'median' feature inclusion index, estimates coefficients for the selected features in parallel for each subset, and then averages these estimates. The algorithm is simple, involves very minimal communication, scales efficiently in both sample and feature size, and has...
We consider problems for which one has incomplete binary matrices that evolve with time (e.g., th... more We consider problems for which one has incomplete binary matrices that evolve with time (e.g., the votes of legislators on particular legislation, with each year characterized by a different such matrix). An objective of such analysis is to infer structure and inter-relationships underlying the matrices, here defined by latent features associated with each axis of the matrix. In addition, it is assumed that documents are available for the entities associated with at least one of the matrix axes. By jointly analyzing the matrices and documents, one may be used to inform the other within the analysis, and the model offers the opportunity to predict matrix values (e.g., votes) based only on an associated document (e.g., legislation). The research presented here merges two areas of machine-learning that have previously been investigated separately: incomplete-matrix analysis and topic modeling. The analysis is performed from a Bayesian perspective, with efficient inference constituted v...
We propose a general scheme for solving convex and non-convex optimization problems on manifolds.... more We propose a general scheme for solving convex and non-convex optimization problems on manifolds. The central idea is that, by adding a multiple of the squared retraction distance to the objective function in question, we "convexify" the objective function and solve a series of convex sub-problems in the optimization procedure. One of the key challenges for optimization on manifolds is the difficulty of verifying the complexity of the objective function, e.g., whether the objective function is convex or non-convex, and the degree of non-convexity. Our proposed algorithm adapts to the level of complexity in the objective function. We show that when the objective function is convex, the algorithm provably converges to the optimum and leads to accelerated convergence. When the objective function is non-convex, the algorithm will converge to a stationary point. Our proposed method unifies insights from Nesterov's original idea for accelerating gradient descent algorithms w...
Although there are many methods for functional data analysis (FDA), little emphasis is put on cha... more Although there are many methods for functional data analysis (FDA), little emphasis is put on characterizing variability among volatilities of individual functions. In particular, certain individuals exhibit erratic swings in their trajectory while other individuals have more stable trajectories. There is evidence of such volatility heterogeneity in blood pressure trajectories during pregnancy, for example, and reason to suspect that volatility is a biologically important feature. Most FDA models implicitly assume similar or identical smoothness of the individual functions, and hence can lead to misleading inferences on volatility and an inadequate representation of the functions. We propose a novel class of FDA models characterized using hierarchical stochastic differential equations. We model the derivatives of a mean function and deviation functions using Gaussian processes, while also allowing covariate dependence including on the volatilities of the deviation functions. Followi...
Models for distributions of shapes contained within images can be widely used in biomedical appli... more Models for distributions of shapes contained within images can be widely used in biomedical applications ranging from tumor tracking for targeted radiation therapy to classifying cells in a blood sample. Our focus is on hierarchical probability models for the shape and size of simply connected 2D closed curves, avoiding the need to specify landmarks through modeling the entire curve while borrowing information across curves for related objects. Prevalent approaches follow a fundamentally dierent strategy in providing an initial
In nonparametric regression, it is common for the inputs to fall in a restricted subset of Euclid... more In nonparametric regression, it is common for the inputs to fall in a restricted subset of Euclidean space. Typical kernel-based methods that do not take into account the intrinsic geometry of the domain across which observations are collected may produce sub-optimal results. In this article, we focus on solving this problem in the context of Gaussian process (GP) models, proposing a new class of Graph based GPs (GL-GPs), which learn a covariance that respects the geometry of the input domain. As the heat kernel is intractable computationally, we approximate the covariance using finitely-many eigenpairs of the Graph Laplacian (GL). The GL is constructed from a kernel which depends only on the Euclidean coordinates of the inputs. Hence, we can benefit from the full knowledge about the kernel to extend the covariance structure to newly arriving samples by a Nyström type extension. We provide substantial theoretical support for the GL-GP methodology, and illustrate performance gains in...
Markov chain Monte Carlo (MCMC) sampling algorithms have dominated the literature on posterior co... more Markov chain Monte Carlo (MCMC) sampling algorithms have dominated the literature on posterior computation. However, MCMC faces substantial hurdles in performing efficient posterior sampling for challenging Bayesian models, particularly in high-dimensional and large data settings. Motivated in part by such hurdles, an intriguing new class of piecewise deterministic Markov processes (PDMPs) has recently been proposed as an alternative to MCMC. One of the most popular types of PDMPs is known as the zig-zag (ZZ) sampler. Such algorithms require a computational upper bound in a Poisson thinning step, with performance improving for tighter bounds. In order to facilitate scaling to larger classes of problems, we propose a general class of Gibbs zig-zag (GZZ) samplers. GZZ allows parameters to be updated in blocks with ZZ applied to certain parameters and traditional MCMC style updates to others. This provides a flexible framework to combine PDMPs with the rich literature on MCMC algorithm...
We consider the problem of flexible modeling of higher order hidden Markov models when the number... more We consider the problem of flexible modeling of higher order hidden Markov models when the number of latent states and the nature of the serial dependence, including the true order, are unknown. We propose Bayesian nonparametric methodology based on tensor factorization techniques that can characterize any transition probability with a specified maximal order, allowing automated selection of the important lags and capturing higher order interactions among the lags. Theoretical results provide insights into identifiability of the emission distributions and asymptotic behavior of the posterior. We design efficient Markov chain Monte Carlo algorithms for posterior computation. In simulation experiments, the method vastly outperformed its first and higher order competitors not just in higher order settings, but, remarkably, also in first order cases. Practical utility is illustrated using real world applications. Some
The dimension of the parameter space is typically unknown in a variety of models that rely on fac... more The dimension of the parameter space is typically unknown in a variety of models that rely on factorizations. For example, in factor analysis the number of latent factors is not known and has to be inferred from the data. Although classical shrinkage priors are useful in such contexts, increasing shrinkage priors can provide a more effective approach that progressively penalizes expansions with growing complexity. In this article we propose a novel increasing shrinkage prior, called the cumulative shrinkage process, for the parameters that control the dimension in overcomplete formulations. Our construction has broad applicability and is based on an interpretable sequence of spike-and-slab distributions which assign increasing mass to the spike as the model complexity grows. Using factor analysis as an illustrative example, we show that this formulation has theoretical and practical advantages relative to current competitors, including an improved ability to recover the model dimens...
Fitting statistical models is computationally challenging when the sample size or the dimension o... more Fitting statistical models is computationally challenging when the sample size or the dimension of the dataset is huge. An attractive approach for down-scaling the problem size is to first partition the dataset into subsets and then fit using distributed algorithms. The dataset can be partitioned either horizontally (in the sample space) or vertically (in the feature space). While the majority of the literature focuses on sample space partitioning, feature space partitioning is more effective when $p\gg n$. Existing methods for partitioning features, however, are either vulnerable to high correlations or inefficient in reducing the model dimension. In this paper, we solve these problems through a new embarrassingly parallel framework named DECO for distributed variable selection and parameter estimation. In DECO, variables are first partitioned and allocated to $m$ distributed workers. The decorrelated subset data within each worker are then fitted via any algorithm designed for hig...
Many modern applications collect large sample size and highly imbalanced categorical data, with s... more Many modern applications collect large sample size and highly imbalanced categorical data, with some categories being relatively rare. Bayesian hierarchical models are well motivated in such settings in providing an approach to borrow information to combat data sparsity, while quantifying uncertainty in estimation. However, a fundamental problem is scaling up posterior computation to massive sample sizes. In categorical data models, posterior computation commonly relies on data augmentation Gibbs sampling. In this article, we study computational efficiency of such algorithms in a large sample imbalanced regime, showing that mixing is extremely poor, with a spectral gap that converges to zero at a rate proportional to the square root of sample size or faster. This theoretical result is verified with empirical performance in simulations and an application to a computational advertising data set. In contrast, algorithms that bypass data augmentation show rapid mixing on the same dataset.
In many applications, there is interest in clustering very high-dimensional data. A common strate... more In many applications, there is interest in clustering very high-dimensional data. A common strategy is first stage dimensionality reduction followed by a standard clustering algorithm, such as k-means. This approach does not target dimension reduction to the clustering objective, and fails to quantify uncertainty. Model-based Bayesian approaches provide an appealing alternative, but often have poor performance in high-dimensions, producing too many or too few clusters. This article provides an explanation for this behavior through studying the clustering posterior in a non-standard setting with fixed sample size and increasing dimensionality. We show that the finite sample posterior tends to either assign every observation to a different cluster or all observations to the same cluster as dimension grows, depending on the kernels and prior specification but not on the true data-generating model. To find models avoiding this pitfall, we define a Bayesian oracle for clustering, with th...
In multivariate data analysis, it is often important to estimate a graph characterizing dependenc... more In multivariate data analysis, it is often important to estimate a graph characterizing dependence among p variables. A popular strategy uses the non-zero entries in a p× p covariance or precision matrix, typically requiring restrictive modeling assumptions for accurate graph recovery. To improve model robustness, we instead focus on estimating the backbone of the dependence graph. We use a spanning tree likelihood, based on a minimalist graphical model that is purposely overly-simplified. Taking a Bayesian approach, we place a prior on the space of trees and quantify uncertainty in the graphical model. In both theory and experiments, we show that this model does not require the population graph to be a spanning tree or the covariance to satisfy assumptions beyond positive-definiteness. The model accurately recovers the backbone of the population graph at a rate competitive with existing approaches but with better robustness. We show combinatorial properties of the spanning tree, wh...
Uploads