article

Posterior consistency in conditional distribution estimation

Authors:

David B. Dunson,

Surya T. TokdarAuthors Info & Claims

Journal of Multivariate Analysis, Volume 116

Pages 456 - 472

https://doi.org/10.1016/j.jmva.2013.01.011

Published: 01 April 2013 Publication History

Abstract

A wide variety of priors have been proposed for nonparametric Bayesian estimation of conditional distributions, and there is a clear need for theorems providing conditions on the prior for large support, as well as posterior consistency. Estimation of an uncountable collection of conditional distributions across different regions of the predictor space is a challenging problem, which differs in some important ways from density and mean regression estimation problems. Defining various topologies on the space of conditional distributions, we provide sufficient conditions for posterior consistency focusing on a broad class of priors formulated as predictor-dependent mixtures of Gaussian kernels. This theory is illustrated by showing that the conditions are satisfied for a class of generalized stick-breaking process mixtures in which the stick-breaking lengths are monotone, differentiable functions of a continuous stochastic process. We also provide a set of sufficient conditions for the case where stick-breaking lengths are predictor independent, such as those arising from a fixed Dirichlet process prior.

References

[1]

Adler, R., An Introduction to Continuity, Extrema, and Related Topics for General Gaussian Processes, Vol. 12. 1990. Institute of Mathematical Statistics.

[2]

Barrientos, F., Jara, A. and Quintana, F., On the support of MacEachern's dependent Dirichlet processes. Bayesian Analysis. v7. 1-34.

[3]

Barron, A., Schervish, M. and Wasserman, L., The consistency of posterior distributions in nonparametric problems. The Annals of Statistics. v27. 536-561.

[4]

Bhattacharya, A. and Dunson, D., Strong consistency of nonparametric Bayes density estimation on compact metric spaces with applications to specific manifolds. Annals of the Institute of Statistical Mathematics. 1-28.

[5]

I. Castillo, R. Nickl, Nonparametric bernstein-von mises theorems, 2012. Preprint arXiv:1208.3862.

[6]

Nonparametric Bayes conditional distribution modeling with variable selection. Journal of the American Statistical Association. v104. 1646-1660.

[7]

Bayesian nonparametric nonproportional hazards survival modeling. Biometrics. v65. 762-771.

[8]

De Iorio, M., Mueller, P., Rosner, G. and MacEachern, S., An ANOVA model for dependent random measures. Journal of the American Statistical Association. v99. 205-215.

[9]

Dunson, D. and Park, J., Kernel stick-breaking processes. Biometrika. v95. 307-323.

[10]

Dunson, D., Pillai, N. and Park, J., Bayesian density regression. Journal of the Royal Statistical Society: Series B. v69. 163-183.

[11]

Fan, J., Yao, Q. and Tong, H., Estimation of conditional densities and sensitivity measures in nonlinear dynamical systems. Biometrika. v83. 189-206.

[12]

Ferguson, T., A Bayesian analysis of some nonparametric problems. The Annals of Statistics. v1. 209-230.

[13]

Ferguson, T., Prior distributions on spaces of probability measures. The Annals of Statistics. v2. 615-629.

[14]

Freedman, D., Wald lecture: on the Bernstein-von Mises theorem with infinite-dimensional parameters. The Annals of Statistics. v27. 1119-1141.

[15]

Gelfand, A., Kottas, A. and MacEachern, S., Bayesian nonparametric spatial modeling with Dirichlet process mixing. Journal of the American Statistical Association. v100. 1021-1035.

[16]

Ghosal, S., Ghosh, J. and Ramamoorthi, R., Posterior consistency of Dirichlet mixtures in density estimation. The Annals of Statistics. v27. 143-158.

[17]

Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities. The Annals of Statistics. v29. 1233-1263.

[18]

Ghosal, S. and van der Vaart, A., Posterior convergence rates of Dirichlet mixtures at smooth densities. The Annals of Statistics. v35. 697-723.

[19]

Griffin, J. and Steel, M., Order-based dependent Dirichlet processes. Journal of the American Statistical Association. v101. 179-194.

[20]

Bayesian nonparametric modelling with the dirichlet process regression smoother. Statistica Sinica. v20. 1507-1527.

[21]

Gibbs sampling methods for stick-breaking priors. Journal of the American Statistical Association. v96. 161-173.

[22]

A split-merge Markov Chain Monte Carlo procedure for the dirichlet process mixture model. Journal of Computational and Graphical Statistics. v13. 158-182.

[23]

Jara, A., Lesaffre, E., De Iorio, M. and Quintana, F., Bayesian semiparametric inference for multivariate doubly-interval-censored data. The Annals of Applied Statistics. v4. 2126-2149.

[24]

Kalli, M., Griffin, J. and Walker, S., Slice sampling mixture models. Statistics and computing. 1-13.

[25]

Kruijer, W., Rousseau, J. and Van Der Vaart, A., Adaptive Bayesian density estimation with location-scale mixtures. Electronic Journal of Statistics. v4. 1225-1257.

[26]

Lo, A., On a class of Bayesian nonparametric estimates: I. Density estimates. The Annals of Statistics. v12. 351-357.

[27]

Dependent nonparametric processes. In: ASA Proceedings of the Section on Bayesian Statistical Science, American Statistical Association, Alexandria, VA. pp. 50-55.

[28]

Minka, T.P., Expectation propagation for approximate Bayesian inference. In: Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, Morgan Kaufmann Publishers Inc. pp. 362-369.

[29]

Müller, P., Erkanli, A. and West, M., Bayesian curve fitting using multivariate normal mixtures. Biometrika. v83. 67-79.

[30]

Norets, A., Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics. v38. 1733-1766.

[31]

A. Norets, J. Pelenis, Posterior consistency in conditional density estimation by covariate dependent mixtures, Unpublished Manuscript, Princeton Univ, 2010.

[32]

O. Papaspiliopoulos, A note on posterior sampling from Dirichlet mixture models, Technical Report, 2008.

[33]

Papaspiliopoulos, O. and Roberts, G., Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models. Biometrika. v95. 169

[34]

Papaspiliopoulos, O. and Roberts, G., Retrospective Markov Chain Monte Carlo methods for Dirichlet process hierarchical models. Biometrika. v95. 169-183.

[35]

D. Pati, D. Dunson, Bayesian nonparametric regression with varying residual density, Unpublished Paper, 2009.

[36]

Rivoirard, V. and Rousseau, J., Bernstein-von Mises theorem for linear functionals of the density. The Annals of Statistics. v40. 1489-1523.

[37]

Rodriguez, A. and Dunson, D., Nonparametric Bayesian models through probit stick-breaking processes. Bayesian Analysis. v6. 145-178.

[38]

A. Rojas, C. Genovese, C. Miller, R. Nichol, L. Wasserman, Conditional density estimation using finite mixture models with an application to astrophysics, 2005.

[39]

Scricciolo, C., Posterior rates of convergence for dirichlet mixtures of exponential power densities. Electronic Journal of Statistics. v5. 270-308.

[40]

W. Shen, S. Tokdar, S. Ghosal, Adaptive Bayesian multivariate density estimation with Dirichlet mixtures, 2011. Preprint arXiv:1109.6406.

[41]

Tang, Y. and Ghosal, S., A consistent nonparametric Bayesian procedure for estimating autoregressive conditional densities. Computational Statistics & Data Analysis. v51. 4424-4437.

[42]

Tang, Y. and Ghosal, S., Posterior consistency of Dirichlet mixtures for estimating a transition density. Journal of Statistical Planning and Inference. v137. 1711-1726.

[43]

Tokdar, S., Posterior consistency of Dirichlet location-scale mixture of normals in density estimation and regression. Sankhy¿: The Indian Journal of Statistics. v67. 90-110.

[44]

S. Tokdar, Adaptive convergence rates in Dirichlet process mixtures of multivariate normals, 2011. Preprint arXiv:1111.4148.

[45]

Posterior consistency of logistic Gaussian process priors in density estimation. Journal of Statistical Planning and Inference. v137. 34-42.

[46]

Bayesian density regression with logistic Gaussian process and subspace projection. Bayesian Analysis. v5. 1-26.

[47]

van der Vaart, A. and van Zanten, J., Reproducing kernel Hilbert spaces of Gaussian priors. IMS Collections. v3. 200-222.

[48]

van der Vaart, A. and van Zanten, J., Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth. The Annals of Statistics. v37. 2655-2675.

[49]

Walker, S., Sampling the Dirichlet mixture model with slices. Communications in Statistics-Simulation and Computation. v36. 45-54.

[50]

Wu, Y. and Ghosal, S., Kullback Leibler property of kernel mixture priors in Bayesian density estimation. Electronic Journal of Statistics. v2. 298-331.

[51]

Wu, Y. and Ghosal, S., The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation. Journal of Multivariate Analysis. 2411-2419.

[52]

J. Yoon, Bayesian analysis of conditional density functions: a limited information approach, Unpublished Manuscript, Claremont Mckenna College, 2009.

Cited By

Barzegar ZRivaz F(2019)A scalable Bayesian nonparametric model for large spatio-temporal dataComputational Statistics10.1007/s00180-019-00905-y35:1(153-173)Online publication date: 12-Jun-2019
https://dl.acm.org/doi/10.1007/s00180-019-00905-y
Shang YDunson DSong J(2017)Exploiting Big Data in Logistics Risk Assessment via Bayesian NonparametricsOperations Research10.1287/opre.2017.161265:6(1574-1588)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1287/opre.2017.1612

Index Terms

Posterior consistency in conditional distribution estimation

Index terms have been assigned to the content through auto-classification.

Recommendations

The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation

Density estimation, especially multivariate density estimation, is a fundamental problem in nonparametric inference. In the Bayesian approach, Dirichlet mixture priors are often used in practice for such problems. However, the asymptotic properties of ...
Importance conditional sampling for Pitman–Yor mixtures
Abstract
Nonparametric mixture models based on the Pitman–Yor process represent a flexible tool for density estimation and clustering. Natural generalization of the popular class of Dirichlet process mixture models, they allow for more robust inference on ...
Bayesian entropy estimation for countable discrete distributions

We consider the problem of estimating Shannon's entropy H from discrete data, in cases where the number of possible symbols is unknown or even countably infinite. The Pitman-Yor process, a generalization of Dirichlet process, provides a tractable prior ...

Comments

Information & Contributors

Information

Published In

cover image Journal of Multivariate Analysis

Journal of Multivariate Analysis Volume 116, Issue

April, 2013

498 pages

ISSN:0047-259X

Issue’s Table of Contents

Copyright © Elsevier Inc. © 2013.

Publisher

Academic Press, Inc.

United States

Publication History

Published: 01 April 2013

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Barzegar ZRivaz F(2019)A scalable Bayesian nonparametric model for large spatio-temporal dataComputational Statistics10.1007/s00180-019-00905-y35:1(153-173)Online publication date: 12-Jun-2019
https://dl.acm.org/doi/10.1007/s00180-019-00905-y
Shang YDunson DSong J(2017)Exploiting Big Data in Logistics Risk Assessment via Bayesian NonparametricsOperations Research10.1287/opre.2017.161265:6(1574-1588)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1287/opre.2017.1612

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents