research-article

PLDA+: Parallel latent dirichlet allocation with data placement and pipeline processing

Authors:

Edward Y. Chang,

Maosong SunAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 2, Issue 3

Article No.: 26, Pages 1 - 18

https://doi.org/10.1145/1961189.1961198

Published: 06 May 2011 Publication History

Abstract

Previous methods of distributed Gibbs sampling for LDA run into either memory or communication bottlenecks. To improve scalability, we propose four strategies: data placement, pipeline processing, word bundling, and priority-based scheduling. Experiments show that our strategies significantly reduce the unparallelizable communication bottleneck and achieve good load balancing, and hence improve scalability of LDA.

References

[1]

Asuncion, A., Smyth, P., and Welling, M. 2008. Asynchronous distributed learning of topic models. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'08). 81--88.

[2]

Asuncion, A., Smyth, P., and Welling, M. 2010. Asynchronous distributed estimation of topic models for document analysis. Statist. Methodol. 8, 1, 3--17.

[3]

Berenbrink, P., Friedetzky, T., Hu, Z., and Martin, R. 2008. On weighted balls-into-bins games. Theor. Comput. Sci. 409, 3, 511--520.

Digital Library

[4]

Blei, D. M., Ng, A. Y., and Jordan, M. I. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993--1022.

[5]

Blinn, J. 1991. A trip down the graphics pipeline: Line clipping. IEEE Comput. Graph. Appl. 11, 1, 98--105.

Digital Library

[6]

Chemudugunta, C., Smyth, P., and Steyvers, M. 2007. Modeling general and specific aspects of documents with a probabilistic topic model. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'07). 241--248.

[7]

Chen, W., Chu, J., Luan, J., Bai, H., Wang, Y., and Chang, E. 2009. Collaborative filtering for orkut communities: Discovery of user latent behavior. In Proceedings of the International World Wide Web Conference (WWW'09). 681--690.

Digital Library

[8]

Chu, C.-T., Kim, S. K., Lin, Y.-A., Yu, Y., Bradski, G., Ng, A. Y., and Olukotun, K. 2006. Mapreduce for machine learning on multicore. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'06).

[9]

Dean, J. and Ghemawat, S. 2004. Mapreduce: Simplified data processing on large clusters. In Proceedings of the ACM USENIX Symposium on Operating Systems Design and Implentation (OSDI'04). 137--150.

Digital Library

[10]

Gomes, R., Welling, M., and Perona, P. 2008. Memory bounded inference in topic models. In Proceedings of the International Conference on Machine Learning (ICML'08). 344--351.

Digital Library

[11]

Graham, S., Snir, M., and Patterson, C. 2005. Getting Up to Speed: The Future of Supercomputing. National Academies Press.

[12]

Griffiths, T. and Steyvers, M. 2004. Finding scientific topics. Proc. Nat. Acad. Sci. United States Amer. 101, 90001, 5228--5235.

[13]

Li, W. and Mccallum, A. 2006. Pachinko allocation: DAG-Structured mixture models of topic correlations. In Proceedings of the International Conference on Machine Learning (ICML'06).

Digital Library

[14]

Mimno, D. M. and Mccallum, A. 2007. Organizing the OCA: Learning faceted subjects from a library of digital books. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries. 376--385.

Digital Library

[15]

Newman, D., Asuncion, A., Smyth, P., and Welling, M. 2007. Distributed inference for latent dirichlet allocation. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'07). 1081--1088.

[16]

Newman, D., Asuncion, A., Smyth, P., and Welling, M. 2009. Distributed algorithms for topic models. J. Mach. Learn. Res. 10, 1801--1828.

Digital Library

[17]

Porteous, I., Newman, D., Ihler, A., Asuncion, A., Smyth, P., and Welling, M. 2008. Fast collapsed gibbs sampling for latent dirichlet allocation. In Proceedings of the International SIGKDD Conference on Knowledge Discovery and Data Mining (KDD'08). 569--577.

Digital Library

[18]

Rosen-Zvi, M., Chemudugunta, C., Griffiths, T., Smyth, P., and Steyvers, M. 2010. Learning author-topic models from text corpora. ACM Trans. Inf. Syst. 28, 1, 1--38.

Digital Library

[19]

Shen, J. P. and Lipasti, M. H. 2005. Modern Processor Design: Fundamentals of Superscalar Processors. McGraw-Hill Higher Education.

[20]

Wang, Y., Bai, H., Stanton, M., Chen, W., and Chang, E. 2009. PLDA: Parallel latent dirichlet allocation for large-scale applications. In Algorithmic Aspects in Information and Management. 301--314.

Digital Library

[21]

Yan, F., Xu, N., and Qi, Y. 2009. Parallel inference for latent dirichlet allocation on graphics processing units. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'09). 2134--2142.

Cited By

Yang HLi JChen S(2024)TopicRefiner: Coherence-Guided Steerable LDA for Visual Topic EnhancementIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.326689030:8(4542-4557)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3266890
Wu XNguyen TLuu A(2024)A survey on neural topic models: methods, applications, and challengesArtificial Intelligence Review10.1007/s10462-023-10661-757:2Online publication date: 25-Jan-2024
https://doi.org/10.1007/s10462-023-10661-7
Zhao KCong GLi X(2022)PGeoTopic: A Distributed Solution for Mining Geographical Topic ModelsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.298914234:2(881-893)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TKDE.2020.2989142
Show More Cited By

Index Terms

PLDA+: Parallel latent dirichlet allocation with data placement and pipeline processing

Recommendations

Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey

Topic modeling is one of the most powerful techniques in text mining for data mining, latent data discovery, and finding relationships among data and text documents. Researchers have published many articles in the field of topic modeling and applied in ...
Generating contextualized sentiment lexica based on latent topics and user ratings
HT '13: Proceedings of the 24th ACM Conference on Hypertext and Social Media

Sentiment lexica are useful for analyzing opinions in Web collections, for domain-dependent sentiment classification, and as sub-components of recommender systems. In this paper, we present a strategy for automatically generating topic-dependent lexica ...
Learning author-topic models from text corpora

We propose an unsupervised learning technique for extracting information about authors and topics from large text collections. We model documents as if they were generated by a two-stage stochastic process. An author is represented by a probability ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 2, Issue 3

April 2011

259 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/1961189

Issue’s Table of Contents

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 May 2011

Accepted: 01 October 2010

Revised: 01 June 2010

Received: 01 April 2010

Published in TIST Volume 2, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

107
Total Citations
View Citations
1,260
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang HLi JChen S(2024)TopicRefiner: Coherence-Guided Steerable LDA for Visual Topic EnhancementIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.326689030:8(4542-4557)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3266890
Wu XNguyen TLuu A(2024)A survey on neural topic models: methods, applications, and challengesArtificial Intelligence Review10.1007/s10462-023-10661-757:2Online publication date: 25-Jan-2024
https://doi.org/10.1007/s10462-023-10661-7
Zhao KCong GLi X(2022)PGeoTopic: A Distributed Solution for Mining Geographical Topic ModelsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.298914234:2(881-893)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TKDE.2020.2989142
Alshanik FApon ADu YHerzog ASafro I(2022)Proactive Query Expansion for Streaming Data Using External Sources2022 IEEE International Conference on Big Data (Big Data)10.1109/BigData55660.2022.10020577(701-708)Online publication date: 17-Dec-2022
https://doi.org/10.1109/BigData55660.2022.10020577
Tang BKang LZhang LGuo FHe H(2021)Collaborative Filtering Recommendation Using Nonnegative Matrix Factorization in GPU-Accelerated Spark PlatformScientific Programming10.1155/2021/88411332021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/8841133
Chauhan UShah A(2021)Topic Modeling Using Latent Dirichlet allocationACM Computing Surveys10.1145/346247854:7(1-35)Online publication date: 17-Sep-2021
https://dl.acm.org/doi/10.1145/3462478
Apishev M(2021)Effective Implementations of Topic Modeling AlgorithmsProgramming and Computer Software10.1134/S036176882107002147:7(483-492)Online publication date: 3-Dec-2021
https://doi.org/10.1134/S0361768821070021
Shao YLi XChen YYu LCui B(2021)Sys-TM: A Fast and General Topic Modeling SystemIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2019.295651833:6(2790-2802)Online publication date: 1-Jun-2021
https://doi.org/10.1109/TKDE.2019.2956518
Yang YLi YTung A(2021)NewsLink: Empowering Intuitive News Search with Knowledge Graphs2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00081(876-887)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00081
Jiang DSong YLian RBao SPeng JHe HWu HZhang CChen L(2021)Familia: A Configurable Topic Modeling Framework for Industrial Text EngineeringDatabase Systems for Advanced Applications10.1007/978-3-030-73200-4_36(516-528)Online publication date: 11-Apr-2021
https://dl.acm.org/doi/10.1007/978-3-030-73200-4_36
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents