article

Free access

Learning transformations for clustering and classification

Authors:

Guillermo SapiroAuthors Info & Claims

The Journal of Machine Learning Research, Volume 16, Issue 1

Pages 187 - 225

Published: 01 January 2015 Publication History

PDF eReader Publisher Site

Abstract

A low-rank transformation learning framework for subspace clustering and classification is proposed here. Many high-dimensional data, such as face images and motion sequences, approximately lie in a union of low-dimensional subspaces. The corresponding subspace clustering problem has been extensively studied in the literature to partition such high-dimensional data into clusters corresponding to their underlying low-dimensional subspaces. Low-dimensional intrinsic structures are often violated for real-world observations, as they can be corrupted by errors or deviate from ideal models. We propose to address this by learning a linear transformation on subspaces using nuclear norm as the modeling and optimization criteria. The learned linear transformation restores a low-rank structure for data from the same subspace, and, at the same time, forces a maximally separated structure for data from different subspaces. In this way, we reduce variations within the subspaces, and increase separation between the subspaces for a more robust subspace clustering. This proposed learned robust subspace clustering framework significantly enhances the performance of existing subspace clustering methods. Basic theoretical results presented here help to further support the underlying framework. To exploit the low-rank structures of the transformed subspaces, we further introduce a fast subspace clustering technique, which efficiently combines robust PCA with sparse modeling. When class labels are present at the training stage, we show this low-rank transformation framework also significantly enhances classification performance. Extensive experiments using public data sets are presented, showing that the proposed approach significantly outperforms state-of-the-art methods for subspace clustering and classification. The learned low cost transform is also applicable to other classification frameworks.

References

[1]

R. Basri and D. W. Jacobs. Lambertian reflectance and linear subspaces. IEEE Trans. on Patt. Anal. and Mach. Intell., 25(2):218-233, 2003.

Digital Library

[2]

M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15:1373-1396, 2003.

Digital Library

[3]

S. Boyd, L. Xiao, and A. Mutapcic. Subgradient method. Notes for EE392o, Stanford University, 2003.

[4]

E. J. Candès, X. Li, Y. Ma, and J. Wright. Robust principal component analysis? J. ACM, 58(3):11:1-11:37, June 2011.

Digital Library

[5]

W. R. Carson, M. Chen, M. R. D. Rodrigues, R. Calderbank, and L. Carin. Communications-inspired projection design with application to compressive sensing. SIAM J. Imaging Sci., 5(4):1185-1212, 2012.

[6]

C. Castillo and D. Jacobs. Using stereo matching for 2-D face recognition across pose. IEEE Trans. on Patt. Anal. and Mach. Intell., 31:2298-2304, 2009.

Digital Library

[7]

G. Chen and G. Lerman. Spectral curvature clustering (SCC). International Journal of Computer Vision, 81(3):317-330, 2009.

Digital Library

[8]

R. Collobert, F. Sinz, J. Weston, and L. Bottou. Large scale transductive svms. J. Mach. Learn. Res., 7:1687-1712, December 2006.

Digital Library

[9]

T. P. Dinh and L. T. H. An. Convex analysis approach to d.c. programming: Theory, algorithms and applications. Acta Mathematica Vietnamica, 22(1):289355, 1997.

[10]

S. C. Douglas, S. Amari, and S. Y. Kung. On gradient adaptation with unit-norm constraints. IEEE Trans. on Signal Processing, 48(6):1843-1847, 2000.

Digital Library

[11]

E. Elhamifar and R. Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE Trans. on Patt. Anal. and Mach. Intell., 2013. To appear.

Digital Library

[12]

M. Fazel. Matrix Rank Minimization with Applications. PhD thesis, Stanford University, 2002.

[13]

D. R. Fuhrmann and B. Liu. An iterative algorithm for locating the minimal eigenvector of a symmetric matrix. In Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, Dallas, TX, 1984.

[14]

A. S. Georghiades, P. N. Belhumeur, and D. J. Kriegman. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. on Patt. Anal. and Mach. Intell., 23(6):643-660, June 2001.

Digital Library

[15]

A. Goh and R. Vidal. Segmenting motions of different types by unsupervised manifold clustering. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., Minneapolis, Minnesota, 2007.

[16]

T. Hastie and P. Y. Simard. Metrics and models for handwritten character recognition. Statistical Science, 13(1):54-65, 1998.

[17]

Z. Jiang, Z. Lin, and L. S. Davis. Learning a discriminative dictionary for sparse coding via label consistent K-SVD. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., Colorado springs, CO, 2011.

Digital Library

[18]

O. Kuybeda, G. A. Frank, A. Bartesaghi, M. Borgnia, S. Subramaniam, and G. Sapiro. A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. Journal of Structural Biology, 181:116-127, 2013.

[19]

G. Liu, Z. Lin, and Y. Yu. Robust subspace segmentation by low-rank representation. In International Conference on Machine Learning, Haifa, Israel, 2010.

Digital Library

[20]

U. Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17(4):395-416, December 2007.

Digital Library

[21]

Y. Ma, H. Derksen, W. Hong, and J. Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE Trans. on Patt. Anal. and Mach. Intell., 29 (9):1546-1562, 2007.

Digital Library

[22]

G. Marsaglia and G. P. H. Styan. When does rank (a + b) = rank(a)+rank(b)? Canad. Math. Bull., 15(3), 1972.

[23]

J. Miao and A. Ben-Israel. On principal angles between subspaces in R_n. Linear Algebra and its Applications, 171(0):81-98, 1992.

[24]

J. Neumann, C. Schnörr, and G. Steidl. Combined SVM-based feature selection and classification. Mach. Learn., 61(1-3):129-150, November 2005.

Digital Library

[25]

Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad. Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. Proc. 27th Asilomar Conference on Signals, Systems and Computers, pages 40-44, Nov. 1993.

[26]

Y. Peng, A. Ganesh, J. Wright, W. Xu, and Y. Ma. RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., San Francisco, USA, 2010.

[27]

Q. Qiu and G. Sapiro. Learning transformations for classification forests. In International Conference on Learning Representations, Banff, Canada, 2014.

[28]

Q. Qiu, V. Patel, P. Turaga, and R. Chellappa. Domain adaptive dictionary learning. In Proc. European Conference on Computer Vision, Florence, Italy, Oct. 2012.

Digital Library

[29]

B. Recht, M. Fazel, and P. A. Parrilo. Guaranteed minimum rank solutions to linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):471-501, 2010.

Digital Library

[30]

S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290:2323-2326, 2000.

[31]

L. K. Saul and S. T. Roweis. An introduction to locally linear embedding. 2000. URL http://www.cs.nyu.edu/~roweis/lle/publications.html.

[32]

X. Shen and Y. Wu. A unified approach to salient object detection via low rank matrix recovery. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., Rhode Island, USA, 2012.

Digital Library

[33]

T. Sim, S. Baker, and M. Bsat. The CMU pose, illumination, and expression (PIE) database. IEEE Trans. on Patt. Anal. and Mach. Intell., 25(12):1615-1618, Dec. 2003.

Digital Library

[34]

M. Soltanolkotabi and E. J. Candes. A geometric analysis of subspace clustering with outliers. The Annals of Statistics, 40(4):2195-2238, 2012.

[35]

M. Soltanolkotabi, E. Elhamifar, and E. J. Candès. Robust subspace clustering. CoRR, abs/1301.2603, 2013. URL http://arxiv.org/abs/1301.2603.

[36]

P. Sprechmann, A. M. Bronstein, and G. Sapiro. Learning efficient sparse and low rank models. CoRR, abs/1212.3631, 2012. URL http://arxiv.org/abs/1212.3631.

[37]

B. K. Sriperumbudur and G. R. G. Lanckriet. A proof of convergence of the concave-convex procedure using zangwill's theory. Neural Computation, 24(6):1391-1407, 2012.

Digital Library

[38]

B. K. Sriperumbudur, D. A. Torres, and G. R. G. Lanckriet. Sparse eigen methods by d.c. programming. In International Conference on Machine Learning, 2007.

Digital Library

[39]

C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision, 9:137-154, 1992.

Digital Library

[40]

M.A. Turk and A.P. Pentland. Face recognition using eigenfaces. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., Maui, Hawaii, 1991.

[41]

R. Vidal. Subspace clustering. Signal Processing Magazine, IEEE, 28(2):52-68, 2011.

[42]

R. Vidal, Yi Ma, and S. Sastry. Generalized principal component analysis (GPCA). In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., Madison, Wisconsin, 2003.

Digital Library

[43]

J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong. Locality-constrained linear coding for image classification. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., San Francisco, USA, 2010.

[44]

Y. Wang and H. Xu. Noisy sparse subspace clustering. In International Conference on Machine Learning, Atlanta, USA, 2013.

Digital Library

[45]

G. A. Watson. Characterization of the subdifferential of some matrix norms. Linear Algebra and Applications, 170:1039-1053, 1992.

[46]

J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma. Robust face recognition via sparse representation. IEEE Trans. on Patt. Anal. and Mach. Intell., 31(2):210-227, 2009.

Digital Library

[47]

J. Yan and M. Pollefeys. A general framework for motion segmentation: independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In Proc. European Conference on Computer Vision, Graz, Austria, 2006.

Digital Library

[48]

A. L. Yuille and A. Rangarajan. The concave-convex procedure. Neural Computation, 4: 915-936, 2003.

Digital Library

[49]

Q. Zhang and B. Li. Discriminative k-SVD for dictionary learning in face recognition. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., San Francisco, CA, 2010.

[50]

T. Zhang, A. Szlam, Y. Wang, and G. Lerman. Hybrid linear modeling via local best-fit flats. International Journal of Computer Vision, 100(3):217-240, 2012.

Digital Library

[51]

Z. Zhang, X. Liang, A. Ganesh, and Y. Ma. TILT: transform invariant low-rank textures. In Proc. Asian conference on Computer vision, Queenstown, New Zealand, 2011.

Digital Library

[52]

X. Zhu and D. Ramanan. Face detection, pose estimation and landmark localization in the wild. In Proc. IEEE Computer Society Conf. on Computer Vision and Patt. Recn., Providence, Rhode Island, 2012.

Digital Library

Cited By

Ahmad MQadri SQadri SSaeed IZareen SIqbal ZAlabrah AAlaghbari HMizanur Rahman S(2022)A Lightweight Convolutional Neural Network Model for Liver Segmentation in Medical DiagnosisComputational Intelligence and Neuroscience10.1155/2022/79543332022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/7954333
Li XChen S(2022)A Concise Yet Effective Model for Non-Aligned Incomplete Multi-View and Missing Multi-Label LearningIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.308689544:10_Part_1(5918-5932)Online publication date: 1-Oct-2022
https://dl.acm.org/doi/10.1109/TPAMI.2021.3086895
Kang PLin ZYang ZFang XBronstein ALi QLiu W(2022)Intra-class low-rank regularization for supervised and semi-supervised cross-modal retrievalApplied Intelligence10.1007/s10489-021-02308-352:1(33-54)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1007/s10489-021-02308-3
Show More Cited By

Index Terms

Learning transformations for clustering and classification
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Classification and regression trees
2. Information systems
  1. Information systems applications
    1. Decision support systems
      1. Expert systems

Recommendations

Sparse subspace clustering with low-rank transformation
Abstract
In order to solve the problem that sparse subspace clustering cannot effectively cluster the dataset under non-independent assumption, this paper proposes the sparse subspace clustering with low-rank transformation, which merges the low-rank ...
Low-rank representation with graph regularization for subspace clustering

In this paper, we propose a low-rank representation method that incorporates graph regularization for robust subspace clustering. We make the assumption that high-dimensional data can be approximated as the union of low-dimensional subspaces of unknown ...
Subspace clustering via structure-enforced dictionary learning
Abstract
Many real world data, which we deal with today, often have very high dimensions. These high-dimensional data can be seen as collections of data points from a union of low-dimensional subspaces. Subspace clustering, one solution to the ...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 16, Issue 1

January 2015

3855 pages

ISSN:1532-4435

EISSN:1533-7928

Editors:
Kevin Murphy
Google
,
Bernhard Schölkopf
MPI for Intelligent Systems

Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 January 2015

Revised: 01 June 2014

Published in JMLR Volume 16, Issue 1

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
313
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ahmad MQadri SQadri SSaeed IZareen SIqbal ZAlabrah AAlaghbari HMizanur Rahman S(2022)A Lightweight Convolutional Neural Network Model for Liver Segmentation in Medical DiagnosisComputational Intelligence and Neuroscience10.1155/2022/79543332022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/7954333
Li XChen S(2022)A Concise Yet Effective Model for Non-Aligned Incomplete Multi-View and Missing Multi-Label LearningIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.308689544:10_Part_1(5918-5932)Online publication date: 1-Oct-2022
https://dl.acm.org/doi/10.1109/TPAMI.2021.3086895
Kang PLin ZYang ZFang XBronstein ALi QLiu W(2022)Intra-class low-rank regularization for supervised and semi-supervised cross-modal retrievalApplied Intelligence10.1007/s10489-021-02308-352:1(33-54)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1007/s10489-021-02308-3
Gong LGuo WYang Y(2018)Active learning support vector machines with low-rank transformationIntelligent Data Analysis10.3233/IDA-17349622:4(701-715)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.3233/IDA-173496
Xiong BLiu QXiong JLi SWang SLiang D(2018)Field-of-Experts Filters Guided Tensor CompletionIEEE Transactions on Multimedia10.1109/TMM.2018.280622520:9(2316-2329)Online publication date: 1-Sep-2018
https://dl.acm.org/doi/10.1109/TMM.2018.2806225
Yin PXin JQi Y(2018)Linear Feature Transform and Enhancement of Classification on Deep Neural NetworkJournal of Scientific Computing10.1007/s10915-018-0666-176:3(1396-1406)Online publication date: 1-Sep-2018
https://dl.acm.org/doi/10.1007/s10915-018-0666-1
Fathy MAlavi AChellappa R(2018)Nonlinear Subspace Feature Enhancement for Image Set ClassificationComputer Vision – ACCV 201810.1007/978-3-030-20870-7_9(142-158)Online publication date: 2-Dec-2018
https://dl.acm.org/doi/10.1007/978-3-030-20870-7_9
Qiu QLezama JBronstein ASapiro G(2018)ForestHash: Semantic Hashing with Shallow Random Forests and Tiny Convolutional NetworksComputer Vision – ECCV 201810.1007/978-3-030-01216-8_27(442-459)Online publication date: 8-Sep-2018
https://dl.acm.org/doi/10.1007/978-3-030-01216-8_27
Xiong JLi SLiu QXu X(2017)Analysis-operator guided simultaneous tensor decomposition and completion2017 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP.2017.8296768(2677-2981)Online publication date: 17-Sep-2017
https://dl.acm.org/doi/10.1109/ICIP.2017.8296768
Fathy MAlavi AChellappa R(2016)Discriminative Log-Euclidean feature learning for sparse representation-based recognition of faces from videosProceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence10.5555/3061053.3061091(3359-3367)Online publication date: 9-Jul-2016
https://dl.acm.org/doi/10.5555/3061053.3061091
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents