research-article

DSANLS: Accelerating Distributed Nonnegative Matrix Factorization via Sketching

Authors:

Nikos Mamoulis,

David W. CheungAuthors Info & Claims

WSDM '18: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining

Pages 450 - 458

https://doi.org/10.1145/3159652.3159662

Published: 02 February 2018 Publication History

Abstract

Nonnegative matrix factorization (NMF) has been successfully applied in different fields, such as text mining, image processing, and video analysis. NMF is the problem of determining two nonnegative low rank matrices U and V, for a given input matrix M, such that m ≈ UV⊥. There is an increasing interest in parallel and distributed NMF algorithms, due to the high cost of centralized NMF on large matrices. In this paper, we propose a distributed sketched alternating nonnegative least squares(DSANLS) framework for NMF, which utilizes a matrix sketching technique to reduce the size of nonnegative least squares subproblems in each iteration for U and V. We design and analyze two different random matrix generation techniques and two subproblem solvers. Our theoretical analysis shows that DSANLS converges to the stationary point of the original NMF problem and it greatly reduces the computational cost in each subproblem as well as the communication cost within the cluster. DSANLS is implemented using MPI for communication, and tested on both dense and sparse real datasets. The results demonstrate the efficiency and scalability of our framework, compared to the state-of-art distributed NMF MPI implementation.

References

[1]

N. Ailon and B. Chazelle. Approximate nearest neighbors and the fast johnson-lindenstrauss transform. In STOC, pages 557--563. ACM, 2006.

Digital Library

[2]

A. B. Chan, V. Mahadevan, and N. Vasconcelos. Generalized stauffer--grimson background subtraction for dynamic scenes. Machine Vision and Applications, 22(5):751--766, 2011.

Digital Library

[3]

M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. Theoretical Computer Science, 312(1):3--15, 2004.

Digital Library

[4]

A. Cichocki and P. Anh-Huy. Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE transactions on fundamentals of electronics, communications and computer sciences, 92(3):708--721, 2009.

[5]

K. L. Clarkson and D. P. Woodruff. Low rank approximation and regression in input sparsity time. In STOC, pages 81--90. ACM, 2013.

Digital Library

[6]

M. E. Daube-Witherspoon and G. Muehllehner. An iterative image space reconstruction algorthm suitable for volume ect. IEEE transactions on medical imaging, 5(2):61--66, 1986.

[7]

J. P. Fairbanks, R. Kannan, H. Park, and D. A. Bader. Behavioral clusters in dynamic graphs. Parallel Computing, 47:38--50, 2015.

Digital Library

[8]

N. Gillis. The why and how of nonnegative matrix factorization. Regularization, Optimization, Kernels, and Support Vector Machines, 12(257), 2014.

[9]

R. M. Gower and P. Richtárik. Randomized iterative methods for linear systems. SIAM Journal on Matrix Analysis and Applications, 36(4):1660--1690, 2015.

[10]

L. Grippo and M. Sciandrone. On the convergence of the block nonlinear gauss--seidel method under convex constraints. Operations research letters, 26(3):127--136, 2000.

Digital Library

[11]

D. Grove, J. Milthorpe, and O. Tardieu. Supporting array programming in x10. In ARRAY, page 38. ACM, 2014.

Digital Library

[12]

N. Guan, D. Tao, Z. Luo, and B. Yuan. Nenmf: an optimal gradient method for nonnegative matrix factorization. IEEE Transactions on Signal Processing, 60(6):2882--2898, 2012.

Digital Library

[13]

K. Kanjani. Parallel non negative matrix factorization for document clustering. CPSC-659 (Parallel and Distributed Numerical Algorithms) course. Texas A&M University, Tech. Rep, 2007.

[14]

R. Kannan, G. Ballard, and H. Park. A high-performance parallel algorithm for nonnegative matrix factorization. In PPoPP, page 9. ACM, 2016.

Digital Library

[15]

R. Kannan, G. Ballard, and H. Park. Mpi-faun: An mpi-based framework for alternating-updating nonnegative matrix factorization. arXiv preprint arXiv:1609.09154, 2016.

[16]

H. Kim and H. Park. Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM journal on matrix analysis and applications, 30(2):713--730, 2008.

Digital Library

[17]

J. Kim, Y. He, and H. Park. Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework. Journal of Global Optimization, 58(2):285--319, 2014.

Digital Library

[18]

J. Kim and H. Park. Fast nonnegative matrix factorization: An active-set-like method and comparisons. SIAM Journal on Scientific Computing, 33(6):3261--3281, 2011.

Digital Library

[19]

I. Kotsia, S. Zafeiriou, and I. Pitas. A novel discriminant non-negative matrix factorization algorithm with applications to facial image characterization problems. IEEE Transactions on Information Forensics and Security, 2(3):588--595, 2007.

Digital Library

[20]

D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788--791, 1999.

[21]

D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. In NIPS, pages 556--562, 2001.

Digital Library

[22]

D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. Rcv1: A new benchmark collection for text categorization research. JMLR, 5(Apr):361--397, 2004.

Digital Library

[23]

R. Liao, Y. Zhang, J. Guan, and S. Zhou. Cloudnmf: a mapreduce implementation of nonnegative matrix factorization for large-scale biological datasets. Genomics, proteomics & bioinformatics, 12(1):48--51, 2014.

[24]

C.-J. Lin. Projected gradient methods for nonnegative matrix factorization. Neural computation, 19(10):2756--2779, 2007.

Digital Library

[25]

C. Liu, H.-c. Yang, J. Fan, L.-W. He, and Y.-M. Wang. Distributed nonnegative matrix factorization for web-scale dyadic data analysis on mapreduce. In WWW, pages 681--690. ACM, 2010.

Digital Library

[26]

Y. Lu, P. Dhillon, D. P. Foster, and L. Ungar. Faster ridge regression via the subsampled randomized hadamard transform. In NIPS, pages 369--377, 2013.

Digital Library

[27]

E. Mej'ıa-Roa, D. Tabas-Madrid, J. Setoain, C. Garc'ıa, F. Tirado, and A. Pascual-Montano. Nmf-mgpu: non-negative matrix factorization on multi-gpu systems. BMC bioinformatics, 16(1):1, 2015.

[28]

X. Meng, J. Bradley, B. Yuvaz, E. Sparks, S. Venkataraman, D. Liu, J. Freeman, D. Tsai, M. Amde, S. Owen, et al. Mllib: Machine learning in apache spark. JMLR, 17(34):1--7, 2016.

Digital Library

[29]

A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4):1574--1609, 2009.

Digital Library

[30]

V. P. Pauca, F. Shahnaz, M. W. Berry, and R. J. Plemmons. Text mining using non-negative matrix factorizations. In SDM, pages 452--456. SIAM, 2004.

[31]

N. Pham and R. Pagh. Fast and scalable polynomial kernels via explicit feature maps. In ACM SIGKDD, pages 239--247. ACM, 2013.

Digital Library

[32]

M. Pilanci and M. J. Wainwright. Iterative hessian sketch: Fast and accurate solution approximation for constrained least-squares. JMLR, pages 1--33, 2015.

Digital Library

[33]

M. Pilanci and M. J. Wainwright. Newton sketch: A linear-time optimization algorithm with linear-quadratic convergence. arXiv preprint arXiv:1505.02250, 2015.

[34]

I. Psorakis, S. Roberts, M. Ebden, and B. Sheldon. Overlapping community detection using bayesian non-negative matrix factorization. Physical Review E, 83(6):066114, 2011.

[35]

Y. Qian, C. Tan, N. Mamoulis, and D. W. Cheung. Dsanls: Accelerating distributed nonnegative matrix factorization via sketching. Technical report, HKU CS, 2017.

[36]

S. A. Robila and L. G. Maciak. A parallel unmixing algorithm for hyperspectral images. In Optics East, pages 63840F--63840F. International Society for Optics and Photonics, 2006.

[37]

R. T. Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877--898, 1976.

[38]

F. Wang and P. Li. Efficient nonnegative matrix factorization with random projections. In SDM, pages 281--292. SIAM, 2010.

[39]

W. Xu, X. Liu, and Y. Gong. Document clustering based on non-negative matrix factorization. In SIGIR, pages 267--273. ACM, 2003.

Digital Library

[40]

J. Yin, L. Gao, and Z. M. Zhang. Scalable nonnegative matrix factorization with block-wise updates. In ECML-PKDD, pages 337--352. Springer, 2014.

Digital Library

[41]

R. Zdunek and A. Cichocki. Non-negative matrix factorization with quasi-newton optimization. In ICAISC, pages 870--879. Springer, 2006.

Digital Library

Cited By

Qian YTan CDing DLi HMamoulis N(2022)Fast and Secure Distributed Nonnegative Matrix FactorizationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.298596434:2(653-666)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TKDE.2020.2985964
Han ROsting BWang DXu Y(2022)Probabilistic methods for approximate archetypal analysisInformation and Inference: A Journal of the IMA10.1093/imaiai/iaac00812:1(466-493)Online publication date: 12-May-2022
https://doi.org/10.1093/imaiai/iaac008

Index Terms

DSANLS: Accelerating Distributed Nonnegative Matrix Factorization via Sketching
1. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

On the HSS iteration methods for positive definite Toeplitz linear systems

We study the HSS iteration method for large sparse non-Hermitian positive definite Toeplitz linear systems, which first appears in Bai, Golub and Ng's paper published in 2003 [Z.-Z. Bai, G.H. Golub, M.K. Ng, Hermitian and skew-Hermitian splitting ...
Accurate Eigenvalues and SVDs of Totally Nonnegative Matrices

We consider the class of totally nonnegative (TN) matrices---matrices all of whose minors are nonnegative. Any nonsingular TN matrix factors as a product of nonnegative bidiagonal matrices. The entries of the bidiagonal factors parameterize the set of ...
Product-type skew-Hermitian triangular splitting iteration methods for strongly non-Hermitian positive definite linear systems

By further generalizing the modified skew-Hermitian triangular splitting iteration methods studied in [L. Wang, Z.-Z. Bai, Skew-Hermitian triangular splitting iteration methods for non-Hermitian positive definite linear systems of strong skew-Hermitian ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '18: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining

February 2018

821 pages

ISBN:9781450355810

DOI:10.1145/3159652

General Chairs:
Yi Chang
Jilin University, Huawei Inc.
,
Chengxiang Zhai
University of Illinois Urbana-Champaign
,
Program Chairs:
Yan Liu
University of Southern California
,
Yoelle Maarek
Amazon

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 February 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

European Union's Horizon 2020 research and innovation programme
Hong Kong Research Grant Council

Conference

WSDM 2018

Sponsor:

WSDM 2018: The Eleventh ACM International Conference on Web Search and Data Mining

February 5 - 9, 2018

CA, Marina Del Rey, USA

Acceptance Rates

WSDM '18 Paper Acceptance Rate 81 of 514 submissions, 16%;

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
212
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Qian YTan CDing DLi HMamoulis N(2022)Fast and Secure Distributed Nonnegative Matrix FactorizationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.298596434:2(653-666)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TKDE.2020.2985964
Han ROsting BWang DXu Y(2022)Probabilistic methods for approximate archetypal analysisInformation and Inference: A Journal of the IMA10.1093/imaiai/iaac00812:1(466-493)Online publication date: 12-May-2022
https://doi.org/10.1093/imaiai/iaac008

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents