article

Free access

Similarity-based clustering by left-stochastic matrix factorization

Editors: Kevin Murphy, Bernhard Schölkopf Authors:

Maryam FazelAuthors Info & Claims

The Journal of Machine Learning Research, Volume 14, Issue 1

Pages 1715 - 1746

Published: 01 January 2013 Publication History

PDF eReader Publisher Site

Abstract

For similarity-based clustering, we propose modeling the entries of a given similarity matrix as the inner products of the unknown cluster probabilities. To estimate the cluster probabilities from the given similarity matrix, we introduce a left-stochastic non-negative matrix factorization problem. A rotation-based algorithm is proposed for the matrix factorization. Conditions for unique matrix factorizations and clusterings are given, and an error bound is provided. The algorithm is particularly efficient for the case of two clusters, which motivates a hierarchical variant for cases where the number of desired clusters is large. Experiments show that the proposed left-stochastic decomposition clustering model produces relatively high within-cluster similarity on most data sets and can match given class labels, and that the efficient hierarchical variant performs surprisingly well.

References

[1]

R. Arora. Group Theoretical Methods in Signal Processing: Learning Similarities, Transformations and Invariants. PhD thesis, Univ. of Wisconsin-Madison, 2009a.

[2]

R. Arora. On learning rotations. Advances in Neural Information Processing Systems (NIPS), 2009b.

[3]

R. Arora and K. Livescu. Kernel CCA for multi-view learning of acoustic features using articulatory measurements. In Symp. on Machine Learning in Speech and Language Processing, 2012.

[4]

R. Arora and K. Livescu. Multi-view CCA-based acoustic features for phonetic recognition across speakers and domains. In Int. Conf. on Acoustics, Speech, and Signal Processing, 2013.

[5]

R. Arora and W. A. Sethares. An efficient and stable algorithm for learning rotations. In Proc. Intl. Conf. Pattern Recognition, 2010.

[6]

R. Arora, M. R. Gupta, A. Kapila, and M. Fazel. Clustering by left-stochastic matrix factorization. Proc. Intl. Conf. Machine Learning (ICML), 2011.

[7]

R. Arora, A. Cotter, K. Livescu, and N. Srebro. Stochastic optimization for PCA and PLS. In 50th Annual Allerton Conference on Communication, Control, and Computing, 2012.

[8]

A. Berman and N. Shaked-Monderer. Completely Positive Matrices. World Scientific, 2003.

[9]

M. W. Berry, M. Browne, A. N. Langville, V. P. Pauca, and R. J. Plemmons. Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics and Data Analysis, 52(1):155 - 173, 2007.

[10]

M. Biggs, A. Ghodsi, and S. Vavasis. Nonnegative matrix factorization via rank-one downdate. Proc. Intl. Conf. Machine Learning (ICML), 2008.

[11]

M. Brown and S. Susstrunk. Multispectral SIFT for scene category recognition. Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), 2011.

[12]

J. Bruna and S. Mallat. Classification with scattering operators. Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), 2011.

[13]

L. Cazzanti, M. R. Gupta, and S. Srivastava. Fusing similarities and Euclidean features with generative classifiers. In Proc. IEEE Conf. on Information Fusion (FUSION), 2009.

[14]

Y. Chen, E. K. Garcia, M. R. Gupta, L. Cazzanti, and A. Rahimi. Similarity-based classification: Concepts and algorithms. JMLR, 2009a.

[15]

Y. Chen, M. R. Gupta, and B. Recht. Learning kernels from indefinite similarities. In Proc. Intl. Conf. Machine Learning (ICML), 2009b.

[16]

N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and J. Kandola. On kernel target alignment. Advances in Neural Information Processing Systems (NIPS), 2001.

[17]

P. Deuflhard and M. Weber. Robust Perron cluster analysis in conformation dynamics. Linear Algebra and Its Applications, 398:161-184, 2005.

[18]

C. Ding, X. He, and H. D. Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. SIAM Conf. Data Mining, 2005.

[19]

C. Ding, T. Li, and M. I. Jordan. Convex and semi-nonnegative matrix factorizations. IEEE Trans. PAMI, 32, 2010.

[20]

J. E. Driskell and T. McDonald. Identification of incomplete networks. Technical Report of Florida Maxima Corporation, 2008.

[21]

C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936.

[22]

I. Färber, S. Günnemann, H.-P. Kriegel, P. Kröoger, E. Müller, E. Schubert, T. Seidl, and A. Zimek. On using class-labels in evaluation of clusterings. Proc. ACM SIGKDD, 2010.

[23]

S. Feng, H. Krim, and I. A. Kogan. 3D face recognition using Euclidean integral invariants signature. Proc. IEEE Workshop Statistical Signal Processing, 2007.

[24]

E. Hanusa, D. Krout, and M. R. Gupta. Clutter rejection by clustering likelihood-based similarities. In Proc. IEEE Conf. on Information Fusion (FUSION), 2011.

[25]

T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer-Verlag, New York, 2nd edition, 2009.

[26]

N.-D. Ho. Nonnegative Matrix Factorizations Algorithms and Applications. PhD thesis, Universite Catholique de Louvain, 2008.

[27]

P. O. Hoyer. Non-negative matrix factorization with sparseness constraints. JMLR, 5:1457-1469, 2004.

[28]

L. Kaufman and P. J. Rousseeuw. Finding Groups in Data. Wiley Series in Applied Probability and Statistics. Wiley, 1990.

[29]

H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistic Quarterly, 2:83-97, 1955.

[30]

G. R. G. Lanckriet, M. Deng, N. Cristianini, M. I. Jordan, and W. S. Noble. Kernel-based data fusion and its application to protein function prediction in yeast. In Proc. of the Pacific Symposium on Biocomputing, 2004.

[31]

D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788-791, 1999.

[32]

D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. Advances in Neural Information Processing Systems (NIPS), 2000.

[33]

Y. Li and A. Ngom. NMF Toolbox Ver. 1.0. cs.uwindsor.ca/~li11112c/nmf.html, 2011.

[34]

P. MacNaughton-Smith, W.T. Williams, M. B. Dale, and L. G. Mockett. Dissimilarity analysis: A new technique of hierarchical sub-division. Nature, 202:1034-5, 1964.

[35]

C. Michelot. A finite algorithm for finding the projection of a point onto the canonical simplex of Rⁿ. J. Opt. Theory Appl., 50:195-200, 1986.

[36]

J. K. Nelson and M. R. Gupta. An EM technique for multiple transmitter localization. 41st Conference on Information Science and Systems, pages 610-615, 2007.

[37]

A. Ng, M. Jordan, and Y. Weiss. On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems (NIPS), 2002.

[38]

D. Niu, J. G. Dy, and M. I. Jordan. Multiple non-redundant spectral clustering views. Proc. Intl. Conf. Machine Learning (ICML), 2010.

[39]

P. Paatero. Least-squares formulation of robust non-negative factor analysis. Chemometrics and Intell. Lab. Sys., 37:23 - 35, 1997.

[40]

P. Paatero. The multilinear engine: A table-driven, least squares program for solving multilinear problems, including the n-way parallel factor analysis model. J. Comp. Graph. Stat., 8(4):854- 888, 1999.

[41]

P. Paatero and U. Tapper. Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values. Environmetrics, 5:111-126, 1994.

[42]

S. Philips, J. Pitton, and L. Atlas. Perceptual feature identification for active sonar echoes. Proc. IEEE OCEANS, 2006.

[43]

S. Selim and M. A. Ismail. K-means type algorithms: a generalized convergence theorem and characterization of local optimality. IEEE Trans. PAMI, 6(1):81-87, 1984.

[44]

G. W. Stewart. Matrix Algorithms, Volume I: Basic Decompositions. SIAM, 1998.

[45]

U. von Luxburg. A tutorial on spectral clustering. Technical Report TR-149, Max Planck Institute for Biological Cybernetics, August 2006.

[46]

M. Weber and S. Kube. Robust Perron cluster analysis for various applications in computational life science. Lecture Notes in Computer Science, 3695:55-66, 2005.

[47]

M. Weber, W. Rungsarityotin, and A. Schliep. Perron cluster analysis and its connection to graph partitioning for noisy data. Zentrum Informationstechnik Berlin Report, 04-39, 2004.

[48]

H. Zha, C. Ding, M. Gu, X. He, and H. Simon. Spectral relaxation for k-means clustering. Advances in Neural Information Processing Systems (NIPS), 2001.

Cited By

Gower RLorenz DWinkler M(2024)A Bregman–Kaczmarz method for nonlinear systems of equationsComputational Optimization and Applications10.1007/s10589-023-00541-987:3(1059-1098)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s10589-023-00541-9
Mair SBoubekki ABrefeld U(2017)Frame-based data factorizationsProceedings of the 34th International Conference on Machine Learning - Volume 7010.5555/3305890.3305919(2305-2313)Online publication date: 6-Aug-2017
https://dl.acm.org/doi/10.5555/3305890.3305919
Chang WTay KLim C(2017)A New Evolving Tree-Based Model with Local Re-learning for Document Clustering and VisualizationNeural Processing Letters10.1007/s11063-017-9597-346:2(379-409)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1007/s11063-017-9597-3
Show More Cited By

Similarity-based clustering by left-stochastic matrix factorization
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning

Recommendations

Document Clustering Based on Spectral Clustering and Non-negative Matrix Factorization
IEA/AIE '08: Proceedings of the 21st international conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems: New Frontiers in Applied Artificial Intelligence

In this paper, we propose a novel non-negative matrix factorization (NMF) to the affinity matrix for document clustering, which enforces non-negativity and orthogonality constraints simultaneously. With the help of orthogonality constraints, this NMF ...
New SVD based initialization strategy for non-negative matrix factorization

We give a new method to determine the rank of the factorization for NMF algorithms.We propose a novel method SVD-NMF to enhance initialization for NMF.The compute process is cheap.Numerical results show that SVD-NMF convergent faster than NNDSVD and ...
Clustering by left-stochastic matrix factorization
ICML'11: Proceedings of the 28th International Conference on International Conference on Machine Learning

We propose clustering samples given their pairwise similarities by factorizing the similarity matrix into the product of a cluster probability matrix and its transpose. We propose a rotation-based algorithm to compute this left-stochastic decomposition (...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 14, Issue 1

January 2013

3717 pages

ISSN:1532-4435

EISSN:1533-7928

Editors:
Kevin Murphy
Google
,
Bernhard Schölkopf
MPI for Intelligent Systems

Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 January 2013

Published in JMLR Volume 14, Issue 1

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
180
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)16

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gower RLorenz DWinkler M(2024)A Bregman–Kaczmarz method for nonlinear systems of equationsComputational Optimization and Applications10.1007/s10589-023-00541-987:3(1059-1098)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s10589-023-00541-9
Mair SBoubekki ABrefeld U(2017)Frame-based data factorizationsProceedings of the 34th International Conference on Machine Learning - Volume 7010.5555/3305890.3305919(2305-2313)Online publication date: 6-Aug-2017
https://dl.acm.org/doi/10.5555/3305890.3305919
Chang WTay KLim C(2017)A New Evolving Tree-Based Model with Local Re-learning for Document Clustering and VisualizationNeural Processing Letters10.1007/s11063-017-9597-346:2(379-409)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1007/s11063-017-9597-3
Zhang YTian YFidge CKelly W(2016)Data-aware task scheduling for all-to-all comparison problems in heterogeneous distributed systemsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2016.04.00893:C(87-101)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1016/j.jpdc.2016.04.008
Radhakrishna VSrinivas CGuruRao CCosta CAparicio M(2014)A modified Gaussian similarity measure for clustering software components and documentsProceedings of the International Conference on Information Systems and Design of Communication10.1145/2618168.2618184(99-104)Online publication date: 16-May-2014
https://dl.acm.org/doi/10.1145/2618168.2618184

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents