article

Free access

Improving prediction from dirichlet process mixtures via enrichment

Authors:

David B. Dunson,

Sonia Petrone, and

Lorenzo TrippaAuthors Info & Claims

The Journal of Machine Learning Research, Volume 15, Issue 1

Pages 1041 - 1071

Published: 01 January 2014 Publication History

Abstract

Flexible covariate-dependent density estimation can be achieved by modelling the joint density of the response and covariates as a Dirichlet process mixture. An appealing aspect of this approach is that computations are relatively easy. In this paper, we examine the predictive performance of these models with an increasing number of covariates. Even for a moderate number of covariates, we find that the likelihood for x tends to dominate the posterior of the latent random partition, degrading the predictive performance of the model. To overcome this, we suggest using a different nonparametric prior, namely an enriched Dirichlet process. Our proposal maintains a simple allocation rule, so that computations remain relatively simple. Advantages are shown through both predictive equations and examples, including an application to diagnosis Alzheimer's disease.

References

[1]

C.E. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Annals of Statistics, 2:1152-1174, 1974.

[2]

D.M. Blei, T.L. Griffths, and M.I. Jordan. The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. Journal of the ACM, 57:1-30, 2010.

[3]

P.J. Brown, N.D. Le, and J.V. Zidek. Inference for a covariance matrix. In P.R. Freeman and A.F.M. Smith, editors, Aspects of Uncertainty. A Tribute to D.V. Lindley, pages 77-92. Wiley, Chichester, 1994.

[4]

Y. Chung and D.B. Dunson. Nonparametric Bayes conditional distribution modeling with variable selection. Journal of the American Statistical Association, 104:1646-1660, 2009.

[5]

B.S. Clarke, E. Fokoué, and H.H. Zhang. Principles and Theory for Data Mining and Machine Learning. Springer Series in Statistics, New York, 2009.

[6]

G. Consonni and P. Veronese. Conditionally reducible natural exponential families and enriched conjugate priors. Scandinavian Journal of Statistics, 28:377-406, 2001.

[7]

D.B. Dunson. Nonparametric Bayes local partition models for random effects. Biometrika, 96:249-262, 2009.

[8]

D.B. Dunson and J.H. Park. Kernel stick-breaking processes. Biometrika, 95:307-323, 2008.

[9]

D.B. Dunson, J. Xue, and L. Carin. The matrix stick breaking process: Flexible Bayes meta analysis. Journal of the American Statistical Association, 103:317-327, 2008.

[10]

D.B. Dunson, S. Petrone, and L. Trippa. Partially hierarchical Dirichlet mixtures for flexible clustering and regression. 2011. Unpublished manuscript.

[11]

S. Efromovich. Conditional density estimation in a regression setting. Annals of Statistics, 35:2504-2535, 2007.

[12]

M.D. Escobar and M. West. Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90:577-588, 1995.

[13]

A.E. Gelfand, A. Kottas, and S.N. MacEachern. Bayesian nonparametric spatial modeling with Dirichlet process mixing. Journal of the American Statistical Association, pages 1021-1035, 2005.

[14]

S. Ghosal. Dirichlet process, related priors, and posterior asymptotics. In N.L. Hjort, C. Holmes, P. Müller, and S.G. Walker, editors, Bayesian Nonparametrics: Principles and Practice. Cambridge University Press, 2010.

[15]

J.E. Griffin and M.F.J. Steel. Order-based dependent Dirichlet processes. Journal of the American Statistical Association, 10:179-194, 2006.

[16]

L.A. Hannah, D.M. Blei, and W.B. Powell. Dirichlet process mixtures of generalized linear models. Journal of Machine Learning Research, 12:1923-1953, 2011.

[17]

C. Kang and S. Ghosal. Clusterwise regression using Dirichlet process mixtures. In A. Sengupta, editor, Advances in Multivariate Statistical Methods, pages 305-325. 2009.

[18]

S.N. MacEachern. Dependent nonparametric processes. In ASA Proceedings of the Section on Bayesian Statistical Science, pages 50-55, Alexandria, VA, 1999. American Statistical Association.

[19]

P. Müller and F.A. Quintana. Random partition models with regression on covariates. Journal of Statistical Planning and Inference, 140:2801-2808, 2010.

[20]

P. Müller, A. Erkanli, and M. West. Bayesian curve fitting using multivariate normal mixtures. Biometrika, 88:67-79, 1996.

[21]

R.M. Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistcs, 9:249-265, 2000.

[22]

A. Norets and J. Pelenis. Bayesian modeling of joint and conditional distributions. Journal of Econometrics, 168:332-346, 2012.

[23]

J.H. Park and D.B. Dunson. Bayesian generalized product partition model. Statistica Sinica, 20:1203-1226, 2010.

[24]

S. Petrone and L. Trippa. Bayesian modeling via nested random partitions. In Proceedings of the International Conference on Complex Data Modelling and Computationally Intensive Statistical Methods, Milan, Italy, 2009. Politecnico di Milano.

[25]

S. Petrone, M. Guindani, and A.E. Gelfand. Hybrid Dirichlet mixture models for functional data. Journal of the Royal Statistical Society, Series B, 71:755-782, 2009.

[26]

L. Ren, L. Du, D.B. Dunson, and L. Carin. The logistic stick-breaking process. Journal of Machine Learning and Research, 12:203-239, 2011.

[27]

A. Rodriguez and D.B. Dunson. Nonparametric Bayesian models through probit stick-breaking processes. Bayesian Analysis, 6:145-178, 2011.

[28]

A. Rodriguez, D.B. Dunson, and A.E. Gelfand. Bayesian nonparametric functional data analysis through density estimation. Biometrika, 96:149-162, 2009.

[29]

D.W. Scott. Multivariate Density Estimation: Theory, Practice, and Visualization. John Wiley & Sons, Inc., Hoboken, NJ, 1992.

[30]

B. Shahbaba and R.M. Neal. Nonlinear models using Dirichlet process mixtures. Journal of Machine Learning Research, 10:1829-1850, 2009.

[31]

S.T. Tokdar. Adaptive convergence rates of a Dirichlet process mixture of multivariate normals. 2011. arXiv:1111.4148 [math.ST].

[32]

S.K. Wade, S. Mongelluzzo, and S. Petrone. An enriched conjugate prior for Bayesian nonparametric inference. Bayesian Analysis, 6:359-386, 2011.

[33]

Y.Wu and S. Ghosal. Kullback Leibler property of kernel mixture priors in Bayesian density estimation. Electronic Journal of Statistics, 2:298-331, 2008.

[34]

Y. Wu and S. Ghosal. The L₁-consistency of Dirichlet mixtures in multivariate density estimation. Journal of Multivariate Analysis, 101:2411-2419, 2010.

Cited By

Lin ALu JXuan JZhu FZhang G(2020)A Causal Dirichlet Mixture Model for Causal Inference from Observational DataACM Transactions on Intelligent Systems and Technology10.1145/337950011:3(1-29)Online publication date: 29-Apr-2020
https://dl.acm.org/doi/10.1145/3379500
Page GQuintana F(2018)Calibrating covariate informed product partition modelsStatistics and Computing10.1007/s11222-017-9777-z28:5(1009-1031)Online publication date: 1-Sep-2018
https://dl.acm.org/doi/10.1007/s11222-017-9777-z
Li XHuan JMatwin SYu SFarooq F(2017)Constructivism LearningProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining10.1145/3097983.3097994(285-294)Online publication date: 13-Aug-2017
https://dl.acm.org/doi/10.1145/3097983.3097994

Index Terms

Improving prediction from dirichlet process mixtures via enrichment
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information systems applications
    1. Decision support systems
      1. Expert systems

Recommendations

Variational Bayesian multinomial logistic Gaussian process classification

The multinomial logistic Gaussian process is a flexible non-parametric model for multi-class classification tasks. These tasks are often involved in solving a pattern recognition problem in real life. In such contexts, the multinomial logistic function (...
Read More
Dirichlet process Gaussian mixture models: choice of the base distribution

In the Bayesian mixture modeling framework it is possible to infer the necessary number of components to model the data and therefore it is unnecessary to explicitly restrict the number of components. Nonparametric mixture models sidestep the problem of ...
Read More
Posterior consistency in conditional distribution estimation

A wide variety of priors have been proposed for nonparametric Bayesian estimation of conditional distributions, and there is a clear need for theorems providing conditions on the prior for large support, as well as posterior consistency. Estimation of ...
Read More

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 15, Issue 1

January 2014

4085 pages

ISSN:1532-4435

EISSN:1533-7928

Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 January 2014

Revised: 01 November 2013

Published in JMLR Volume 15, Issue 1

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
159
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)14

Other Metrics

View Author Metrics

Citations

Cited By

Lin ALu JXuan JZhu FZhang G(2020)A Causal Dirichlet Mixture Model for Causal Inference from Observational DataACM Transactions on Intelligent Systems and Technology10.1145/337950011:3(1-29)Online publication date: 29-Apr-2020
https://dl.acm.org/doi/10.1145/3379500
Page GQuintana F(2018)Calibrating covariate informed product partition modelsStatistics and Computing10.1007/s11222-017-9777-z28:5(1009-1031)Online publication date: 1-Sep-2018
https://dl.acm.org/doi/10.1007/s11222-017-9777-z
Li XHuan JMatwin SYu SFarooq F(2017)Constructivism LearningProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining10.1145/3097983.3097994(285-294)Online publication date: 13-Aug-2017
https://dl.acm.org/doi/10.1145/3097983.3097994

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents