article

Free access

Distance Dependent Chinese Restaurant Processes

Authors:

Peter I. FrazierAuthors Info & Claims

The Journal of Machine Learning Research, Volume 12

Pages 2461 - 2488

Published: 01 November 2011 Publication History

Abstract

We develop the distance dependent Chinese restaurant process, a flexible class of distributions over partitions that allows for dependencies between the elements. This class can be used to model many kinds of dependencies between data in infinite clustering models, including dependencies arising from time, space, and network connectivity. We examine the properties of the distance dependent CRP, discuss its connections to Bayesian nonparametric mixture models, and derive a Gibbs sampler for both fully observed and latent mixture settings. We study its empirical performance with three text corpora. We show that relaxing the assumption of exchangeability with distance dependent CRPs can provide a better fit to sequential data and network data. We also show that the distance dependent CRP representation of the traditional CRP mixture leads to a faster-mixing Gibbs sampling algorithm than the one based on the original formulation.

References

[1]

A. Ahmed and E. Xing. Dynamic non-parametric mixture models and the recurrent Chinese restaurant process with applications to evolutionary clustering. In International Conference on Data Mining, 2008.

[2]

C. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. The Annals of Statistics, 2(6):1152-1174, 1974.

[3]

D. Blackwell. Discreteness of Ferguson selections. The Annals of Statistics, 1(2):356-358, 1973.

[4]

D. Blei and P. Frazier. Distance dependent Chinese restaurant processes. In International Conference on Machine Learning, 2010.

[5]

D. Blei and M. Jordan. Variational inference for Dirichlet process mixtures. Journal of Bayesian Analysis, 1(1):121-144, 2005.

[6]

D. Blei, T. Griffiths, and M. Jordan. The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. Journal of the ACM, 57(2):1-30, 2010.

[7]

D.B. Dahl. Distance-based probability distribution for set partitions with applications to Bayesian nonparametrics. In JSM Proceedings. Section on Bayesian Statistical Science, American Statistical Association, Alexandria, Va, 2008.

[8]

H. Daume. Fast search for Dirichlet process mixture models. In Artificial Intelligence and Statistics, San Juan, Puerto Rico, 2007. URL http://pub.hal3.name/#daume07astar-dp.

[9]

J. Duan, M. Guindani, and A. Gelfand. Generalized spatial Dirichlet process models. Biometrika, 94:809-825, 2007.

[10]

D. Dunson. Bayesian dynamic modeling of latent trait distributions. Biostatistics, 2006.

[11]

D. Dunson, N. Pillai, and J. Park. Bayesian density regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):163-183, 2007.

[12]

M. Escobar and M. West. Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90:577-588, 1995.

[13]

T. Ferguson. A Bayesian analysis of some nonparametric problems. The Annals of Statistics, 1: 209-230, 1973.

[14]

E. Fox, E. Sudderth, M. Jordan, and A. Willsky. Developing a tempered HDP-HMM for systems with state persistence. Technical report, MIT Laboratory for Information and Decision Systems, 2007.

[15]

C. Geyer and E. Thompson. Constrained Monte Carlo maximum likelihood for dependent data. Journal of the American Statistical Association, 54(657-699), 1992.

[16]

S. Goldwater, T. Griffiths, and M. Johnson. Interpolating between types and tokens by estimating power-law generators. In Neural Information Processing Systems, 2006.

[17]

J. Griffin and M. Steel. Order-based dependent Dirichlet processes. Journal of the American Statistical Association, 101(473):179-194, 2006.

[18]

J.A. Hartigan. Partition models. Communications in Statistics-Theory and Methods, 19(8):2745- 2756, 1990.

[19]

M. Johnson, T. Griffiths, and Goldwater S. Adaptor grammars: A framework for specifying compositional nonparametric Bayesian models. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 641-648, Cambridge, MA, 2007. MIT Press.

[20]

R. Kass and A. Raftery. Bayes factors. Journal of the American Statistical Association, 90(430): 773-795, 1995.

[21]

W. Li, D. Blei, and A. McCallum. Nonparametric Bayes pachinko allocation. In The 23rd Conference on Uncertainty in Artificial Intelligence, 2007.

[22]

P. Liang, M. Jordan, and B. Taskar. A permutation-augmented sampler for DP mixture models. In International Conference on Machine Learning, 2007.

[23]

S. MacEachern. Dependent nonparametric processes. In ASA Proceedings of the Section on Bayesian Statistical Science, 1999.

[24]

A. McCallum, K. Nigam, J. Rennie, and K. Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 2000.

[25]

K.T. Miller, T.L. Griffiths, and M.I. Jordan. The phylogenetic indian buffet process: A nonexchangeable nonparametric prior for latent features. In David A. McAllester and Petri Myllymaki, editors, UAI, pages 403-410. AUAI Press, 2008.

[26]

P. Mueller and F. Quintana. Random partition models with regression on covariates. In International Conference on Interdisciplinary Mathematical and Statistical Techniques, 2008.

[27]

P. Muller, F. Quintana, and G. Rosner. Bayesian clustering with regression. Working paper, 2008.

[28]

R. Neal. Probabilistic inference using Markov chain Monte Carlo methods. Technical Report CRG-TR- 93-1, Department of Computer Science, University of Toronto, 1993.

[29]

R. Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9(2):249-265, 2000.

[30]

J. Pitman. Combinatorial Stochastic Processes. Lecture Notes for St. Flour Summer School. Springer-Verlag, New York, NY, 2002.

[31]

C. Rasmussen and Z. Ghahramani. Infinite mixtures of Gaussian process experts. In T. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, Cambridge, MA, 2002. MIT Press.

[32]

C. Ritter and M. Tanner. Facilitating the Gibbs sampler: The Gibbs stopper and the Griddy-Gibbs sampler. Journal of the American Statistical Association, 87(419):861-868, 1992.

[33]

C. Robert and G. Casella. Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer-Verlag, New York, NY, 2004.

[34]

E. Sudderth, A. Torralba, W. Freeman, and A. Willsky. Describing visual scenes using transformed Dirichlet processes. In Advances in Neural Information Processing Systems 18, 2005.

[35]

E.B. Sudderth and M. I. Jordan. Shared segmentation of natural scenes using dependent pitman-yor processes. In Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Leon Bottou, editors, NIPS, pages 1585-1592. MIT Press, 2008.

[36]

Y. Teh. A hierarchical Bayesian language model based on Pitman-Yor processes. In Proceedings of the Association of Computational Linguistics, 2006.

[37]

Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476):1566-1581, 2006.

[38]

E. Xing, M. Jordan, and R. Sharan. Bayesian haplotype inference via the Dirichlet process. Journal of Computational Biology, 14(3):267-284, 2007.

[39]

Y. Xue, D. Dunson, and L. Carin. The matrix stick-breaking process for flexible multi-task learning. In International Conference on Machine Learning, 2007.

[40]

X. Zhu, Z. Ghahramani, and J. Lafferty. Time-sensitive Dirichlet process mixture models. Technical Report CMU-CALD-05-104, Carnegie Mellon University, 2005.

Cited By

Mo JGong Z(2023)Cross-City Multi-Granular Adaptive Transfer Learning for Traffic Flow PredictionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.323218535:11(11246-11258)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TKDE.2022.3232185
Martín-Doñas JArzelus HÁlvarez AArellano J(2023)An Online Diarization Approach for Streaming Applications Based on Tree-Clustering and Bayesian ResegmentationText, Speech, and Dialogue10.1007/978-3-031-40498-6_23(258-269)Online publication date: 4-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-40498-6_23
Hosseini SRabiee HHafez HSoltani-Farani A(2022)Classifying a Stream of Infinite Concepts: A Bayesian Non-parametric ApproachMachine Learning and Knowledge Discovery in Databases10.1007/978-3-662-44848-9_1(1-16)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1007/978-3-662-44848-9_1
Show More Cited By

Distance Dependent Chinese Restaurant Processes
1. Computing methodologies

Recommendations

Fast Search for Distance Dependent Chinese Restaurant Processes
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide Web

The distance dependent Chinese Restaurant Processes(dd-CRP), a nonparametric Bayesian model, can model distance sensitive data. Existing inference algorithms for dd-CRP, such as Markov Chain Monte Carlo (MCMC) and variational algorithms, are inefficient ...
Distance dependent Chinese restaurant processes
ICML'10: Proceedings of the 27th International Conference on International Conference on Machine Learning

We develop the distance dependent Chinese restaurant process (CRP), a flexible class of distributions over partitions that allows for non-exchangeability. This class can be used to model dependencies between data in infinite clustering models, including ...
Variational inference for sequential distance dependent chinese restaurant process
ICML'14: Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32

Recently proposed distance dependent Chinese Restaurant Process (ddCRP) generalizes extensively used Chinese Restaurant Process (CRP) by accounting for dependencies between data points. Its posterior is intractable and so far only MCMC methods were used ...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 12, Issue

2/1/2011

3426 pages

ISSN:1532-4435

EISSN:1533-7928

Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 November 2011

Published in JMLR Volume 12

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
694
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)11

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mo JGong Z(2023)Cross-City Multi-Granular Adaptive Transfer Learning for Traffic Flow PredictionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.323218535:11(11246-11258)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TKDE.2022.3232185
Martín-Doñas JArzelus HÁlvarez AArellano J(2023)An Online Diarization Approach for Streaming Applications Based on Tree-Clustering and Bayesian ResegmentationText, Speech, and Dialogue10.1007/978-3-031-40498-6_23(258-269)Online publication date: 4-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-40498-6_23
Hosseini SRabiee HHafez HSoltani-Farani A(2022)Classifying a Stream of Infinite Concepts: A Bayesian Non-parametric ApproachMachine Learning and Knowledge Discovery in Databases10.1007/978-3-662-44848-9_1(1-16)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1007/978-3-662-44848-9_1
Nakano MFujiwara YKimura AYamada TUeda NRanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Permuton-induced Chinese restaurant processProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3542382(27695-27708)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3542382
Abououf MSingh SOtrok HMizouni RDamiani E(2021)Machine Learning in Mobile Crowd Sourcing: A Behavior-Based Recruitment ModelACM Transactions on Internet Technology10.1145/345116322:1(1-28)Online publication date: 9-Nov-2021
https://dl.acm.org/doi/10.1145/3451163
Ladd AChan KNguyen SCadena JNg B(2021)End-to-End Framework for Imputation and State Discovery in Longitudinal Energy DataProceedings of the Twelfth ACM International Conference on Future Energy Systems10.1145/3447555.3466588(475-482)Online publication date: 22-Jun-2021
https://dl.acm.org/doi/10.1145/3447555.3466588
Li HLi HBhowmick S(2020)BRUNCH: Branching Structure Inference of Hybrid Multivariate Hawkes Processes with Application to Social MediaAdvances in Knowledge Discovery and Data Mining10.1007/978-3-030-47426-3_43(553-566)Online publication date: 11-May-2020
https://dl.acm.org/doi/10.1007/978-3-030-47426-3_43
Mehrotra RLalmas MKenney DLim-Meng THashemian G(2019)Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate RecommendationsThe World Wide Web Conference10.1145/3308558.3313613(1256-1267)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313613
Xuan JLu JZhang G(2019)A Survey on Bayesian Nonparametric LearningACM Computing Surveys10.1145/329104452:1(1-36)Online publication date: 25-Jan-2019
https://dl.acm.org/doi/10.1145/3291044
Zaheer MAhmed AWang YSilva DNajork MWu YSanan SChatterjee SCulpepper JMoffat ABennett PLerman K(2019)Uncovering Hidden Structure in Sequence Data via Threading Recurrent ModelsProceedings of the Twelfth ACM International Conference on Web Search and Data Mining10.1145/3289600.3291036(186-194)Online publication date: 30-Jan-2019
https://dl.acm.org/doi/10.1145/3289600.3291036
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents