Distributed-Memory Parallel JointNMF
Authors: 
Srinivas Eswar, Benjamin Cobb, Koby Hayashi, Ramakrishnan Kannan, 
Grey Ballard, 
Richard Vuduc, and Haesun ParkAuthors Info & Claims
Srinivas Eswar, Benjamin Cobb, Koby Hayashi, Ramakrishnan Kannan, Grey Ballard, Richard Vuduc, and Haesun ParkAuthors Info & ClaimsJune 2023
Pages 301 - 312
Abstract
Joint Nonnegative Matrix Factorization (JointNMF) is a hybrid method for mining information from datasets that contain both feature and connection information. We propose distributed-memory parallelizations of three algorithms for solving the JointNMF problem based on Alternating Nonnegative Least Squares, Projected Gradient Descent, and Projected Gauss-Newton. We extend well-known communication-avoiding algorithms using a single processor grid case to our coupled case on two processor grids. We demonstrate the scalability of the algorithms on up to 960 cores (40 nodes) with 60% parallel efficiency. The more sophisticated Alternating Nonnegative Least Squares (ANLS) and Gauss-Newton variants outperform the first-order gradient descent method in reducing the objective on large-scale problems. We perform a topic modelling task on a large corpus of academic papers that consists of over 37 million paper abstracts and nearly a billion citation relationships, demonstrating the utility and scalability of the methods.
References
[1]
Hussam Al Daas, Grey Ballard, Laura Grigori, Suraj Kumar, and Kathryn Rouse. 2022. Brief Announcement: Tight Memory-Independent Parallel Matrix Multiplication Communication Lower Bounds. In Proceedings of the 34th ACM Symposium on Parallelism in Algorithms and Architectures (Philadelphia, PA, USA) (SPAA '22). Association for Computing Machinery, New York, NY, USA, 445--448.
[2]
Dimitri P Bertsekas. 1982. Projected Newton methods for optimization problems with simple constraints. SIAM Journal on control and Optimization 20, 2 (1982), 221--246.
[3]
Dimitri P Bertsekas. 1999. Nonlinear Programming. Athena Scientific.
[4]
Ernie Chan, Marcel Heimlich, Avi Purkayastha, and Robert van de Geijn. 2007. Collective communication: theory, practice, and experience. Concurrency and Computation: Practice and Experience 19, 13 (2007), 1749--1783.
[5]
James Demmel, David Eliahu, Armando Fox, Shoaib Kamil, Benjamin Lipshitz, Oded Schwartz, and Omer Spillinger. 2013. Communication-optimal parallel recursive rectangular matrix multiplication. In 2013 IEEE 27th International Symposium on Parallel and Distributed Processing. IEEE, IEEE, 261--272.
[6]
Rundong Du. 2018. Nonnegative matrix factorization for text, graph, and hybrid data analytics. Ph. D. Dissertation. Georgia Institute of Technology. http://hdl.handle.net/1853/59914
[7]
Rundong Du, Barry Drake, and Haesun Park. 2019. Hybrid clustering based on content and connection structure using joint nonnegative matrix factorization. Journal of Global Optimization 74, 4 (2019), 861--877.
[8]
Srinivas Eswar, Koby Hayashi, Grey Ballard, Ramakrishnan Kannan, Michael A. Matheson, and Haesun Park. 2021. PLANC: Parallel Low-Rank Approximation with Nonnegativity Constraints. ACM Trans. Math. Softw. 47, 3, Article 20 (jun 2021), 37 pages.
[9]
Srinivas Eswar, Koby Hayashi, Grey Ballard, Ramakrishnan Kannan, Richard Vuduc, and Haesun Park. 2020. Distributed-Memory Parallel Symmetric Nonnegative Matrix Factorization. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Atlanta, Georgia) (SC '20). IEEE Press, Article 74, 14 pages.
[10]
Srinivas Eswar, Ramakrishnan Kannan, Richard Vuduc, and Haesun Park. 2021. ORCA: Outlier detection and Robust Clustering for Attributed graphs. Journal of Global Optimization 81, 4 (2021), 967--989.
[11]
Edgar Gabriel, Graham E. Fagg, George Bosilca, Thara Angskun, Jack J. Dongarra, Jeffrey M. Squyres, Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, Andrew Lumsdaine, Ralph H. Castain, David J. Daniel, Richard L. Graham, and Timothy S. Woodall. 2004. OpenMPI: Goals, Concept, and Design of a Next Generation MPI Implementation. In Proceedings, 11th European PVM/MPI Users' Group Meeting. Springer Berlin Heidelberg, Budapest, Hungary, 97--104.
[12]
Nicolas Gillis. 2020. Nonnegative Matrix Factorization. SIAM, Philadelphia, PA.
[13]
Ramakrishnan Kannan, Grey Ballard, and Haesun Park. 2016. A high-performance parallel algorithm for nonnegative matrix factorization. ACM SIGPLAN Notices 51, 8 (2016), 1--11.
[14]
R. Kannan, G. Ballard, and H. Park. 2018. MPI-FAUN: An MPI-Based Framework for Alternating-Updating Nonnegative Matrix Factorization. IEEE Transactions on Knowledge and Data Engineering 30, 3 (March 2018), 544--558.
[15]
Dongmin Kim, Suvrit Sra, and Inderjit S Dhillon. 2007. Fast Newton-type methods for the least squares nonnegative matrix approximation problem. In Proceedings of the 2007 SIAM international conference on data mining. SIAM, 343--354.
[16]
Jingu Kim, Yunlong He, and Haesun Park. 2014. Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework. Journal of Global Optimization 58, 2 (2014), 285--319.
[17]
Da Kuang, Chris Ding, and Haesun Park. 2012. Symmetric nonnegative matrix factorization for graph clustering. In Proceedings of the 2012 SIAM international conference on data mining. SIAM, 106--117.
[18]
Da Kuang, Sangwoon Yun, and Haesun Park. 2015. SymNMF: nonnegative low-rank approximation of a similarity matrix for graph clustering. Journal of Global Optimization 62, 3 (2015), 545--574.
[19]
Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. 2020. Mining of massive datasets (3rd ed. ed.). Cambridge University Press.
[20]
Chih-Jen Lin. 2007. Projected gradient methods for nonnegative matrix factorization. Neural computation 19, 10 (2007), 2756--2779.
[21]
Jialu Liu, Chi Wang, Jing Gao, and Jiawei Han. 2013. Multi-view clustering via joint nonnegative matrix factorization. In Proceedings of the 2013 SIAM international conference on data mining. SIAM, 252--260.
[22]
Ninghao Liu, Xiao Huang, and Xia Hu. 2017. Accelerated Local Anomaly Detection via Resolving Attributed Networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (Melbourne, Australia) (IJCAI'17). AAAI Press, 2337--2343.
[23]
Jorge Nocedal and Stephen J. Wright. 2006. Numerical Optimization (2e ed.). Springer, New York, NY, USA.
[24]
PACE. 2017. Partnership for an Advanced Computing Environment (PACE). https://www.pace.gatech.edu
[25]
Zhen Peng, Minnan Luo, Jundong Li, Huan Liu, and Qinghua Zheng. 2018. ANOMALOUS: A Joint Modeling Approach for Anomaly Detection on Attributed Networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (Stockholm, Sweden) (IJCAI'18). AAAI Press, 3513--3519.
[26]
Ning Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural networks 12, 1 (1999), 145--151.
[27]
Sebastian Ruder. 2017. An overview of gradient descent optimization algorithms. arXiv:1609.04747 [cs.LG]
[28]
Conrad Sanderson and Ryan Curtin. 2016. Armadillo: a template-based C++ library for linear algebra. Journal of Open Source Software 1, 2 (2016), 26.
[29]
Mark Schmidt, Dongmin Kim, and Suvrit Sra. 2012. Projected Newton-type Methods in Machine Learning. In Optimization for Machine Learning, Suvrit Sra, Sebastian Nowozin, and Stephen J Wright (Eds.). MIT Press, Chapter 11, 305--329.
[30]
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MAS) and Applications. In Proceedings of the 24th International Conference on World Wide Web (Florence, Italy) (WWW '15 Companion). Association for Computing Machinery, New York, NY, USA, 243--246.
[31]
Jiliang Tang, Xufei Wang, and Huan Liu. 2011. Integrating social media data for community detection. In Modeling and Mining Ubiquitous Social Media. Springer, 1--20.
[32]
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. Arnet-Miner: Extraction and Mining of Academic Social Networks. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Las Vegas, Nevada, USA) (KDD '08). Association for Computing Machinery, New York, NY, USA, 990--998.
[33]
N Vervliet and L De Lathauwer. 2019. Numerical optimization-based algorithms for data fusion. In Data Handling in Science and Technology. Vol. 31. Elsevier, 81--128.
[34]
Joyce Jiyoung Whang, Rundong Du, Sangwon Jung, Geon Lee, Barry Drake, Qingqing Liu, Seonggoo Kang, and Haesun Park. 2020. MEGA: Multi-view semi-supervised clustering of hypergraphs. Proceedings of the VLDB Endowment 13, 5 (2020), 698--711.
[35]
Zhang Xianyi, Wang Qian, and Zaheer Chothia. 2012. OpenBLAS.
[36]
Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang, Bin Shao, Rui Li, and Kuansan Wang. 2019. OAG: Toward Linking Large-Scale Heterogeneous Entity Graphs. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Anchorage, AK, USA) (KDD '19). Association for Computing Machinery, New York, NY, USA, 2585--2595.
Index Terms
- Distributed-Memory Parallel JointNMF
Recommendations
Highly Scalable Parallel Algorithms for Sparse Matrix Factorization
In this paper, we describe scalable parallel algorithms for symmetric sparse matrix factorization, analyze their performance and scalability, and present experimental results for up to 1,024 processors on a Cray T3D parallel computer. Through our ...
Distributed-memory parallel symmetric nonnegative matrix factorization
SC '20: Proceedings of the International Conference for High Performance Computing, Networking, Storage and AnalysisWe develop the first distributed-memory parallel implementation of Symmetric Nonnegative Matrix Factorization (SymNMF), a key data analytics kernel for clustering and dimensionality reduction. Our implementation includes two different algorithms for ...
Comments
Information & Contributors
Information
Published In
June 2023
505 pages
ISBN:9798400700569
DOI:10.1145/3577193
- Chair:
- Kyle Gallivan,
- Co-chair:
- Efstratios Gallopoulos,
- Program Co-chairs:
- Dimitrios S. Nikolopoulos,
- Ramon Beivide
Copyright © 2023 ACM.
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 June 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICS '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 629 of 2,180 submissions, 29%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 151Total Downloads
- Downloads (Last 12 months)125
- Downloads (Last 6 weeks)4
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in