research-article

Structure-Driven Representation Learning for Deep Clustering

Authors:

Jian YuAuthors Info & Claims

ACM Transactions on Knowledge Discovery from Data, Volume 18, Issue 1

Article No.: 31, Pages 1 - 25

https://doi.org/10.1145/3623400

Published: 16 October 2023 Publication History

Abstract

As an important branch of unsupervised learning methods, clustering makes a wide contribution in the area of data mining. It is well known that capturing the group-discriminative properties of each sample for clustering is crucial. Among them, deep clustering delivers promising results due to the strong representational power of neural networks. However, most of them adopt sample-level learning strategies, and the standalone data point barely captures its holistic cluster’s context and may undergo sub-optimal cluster assignment. To tackle this issue, we propose a Structure-driven Representation Learning (SRL) method by introducing latent structure information into the representation learning process at both the local and global levels. Specifically, a local-structure-driven sample representation strategy is proposed to approximate the estimation of data distribution, which models the neighborhood distribution of samples with potential structure information and exploits statistical dependencies between them to improve cluster consistency. A global-structure-driven cluster representation strategy is designed, where the context of each cluster is sufficiently encoded according to its samples (exemplar-theory) and corresponding prototype (prototype-theory). In this case, each cluster can only be related to its most similar samples, and different clusters are separated as much as possible. These two models are seamlessly combined into a joint optimization problem, which can be efficiently solved. Experiments on six widely-used datasets demonstrate the superiority of SRL over state-of-the-art clustering methods.

References

[1]

Mahdi Abavisani, Alireza Naghizadeh, Dimitris N. Metaxas, and Vishal M. Patel. 2020. Deep subspace clustering with data augmentation. In NeurIPS.

[2]

Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. 2019. Self-labelling via simultaneous clustering and representation. In International Conference on Learning Representations.

[3]

Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. 2022. Efficient deep embedded subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1–10.

[4]

Qi Cai, Yu Wang, Yingwei Pan, Ting Yao, and Tao Mei. 2020. Joint contrastive learning with infinite possibilities. Advances in Neural Information Processing Systems 33 (2020), 12638–12648.

[5]

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV’18). 132–149.

Digital Library

[6]

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems 33 (2020), 9912–9924.

[7]

Jianlong Chang, Yiwen Guo, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. 2019. Deep discriminative clustering analysis. arXiv:1905.01681.

[8]

Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. 2017. Deep adaptive image clustering. In Proceedings of the IEEE International Conference on Computer Vision. 5879–5887.

[9]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning. PMLR, 1597–1607.

[10]

Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. 2020. Debiased contrastive learning. Advances in Neural Information Processing Systems 33 (2020), 8765–8775.

[11]

Adam Coates, Andrew Ng, and Honglak Lee. 2011. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 215–223.

[12]

Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. 2021. Nearest neighbor matching for deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13693–13702.

[13]

Terrance DeVries and Graham W. Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).

[14]

Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. 2016. Deep unsupervised clustering with gaussian mixture variational autoencoders. In International Conference on Learning Representations.

[15]

Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. 2017. Improved deep embedded clustering with local structure preservation. In Ijcai. 1753–1759.

[16]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9729–9738.

[17]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.

[18]

Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. 2017. Learning discrete representations via information maximizing self-augmented training. In Proceedings of the International Conference on Machine Learning. PMLR, 1558–1567.

[19]

Jiabo Huang and Shaogang Gong. 2021. Deep clustering by semantic contrastive learning. arXiv: 2103.02662.

[20]

Jiabo Huang, Shaogang Gong, and Xiatian Zhu. 2020. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8849–8858.

[21]

Anil K. Jain. 2010. Data clustering: 50 years beyond K-means. Pattern Recognition Letters 31, 8 (2010), 651–666.

Digital Library

[22]

Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. 2017. Deep subspace clustering networks. Advances in neural information processing systems.

[23]

Xu Ji, Joao F. Henriques, and Andrea Vedaldi. 2018. Invariant information distillation for unsupervised image segmentation and clustering. arXiv: 1807.06653.

[24]

Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. 2016. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. 1965–1972.

[25]

Alex Krizhevsky, Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases1, 4 (2009).

[26]

Ya Le and Xuan Yang. 2015. Tiny imagenet visual recognition challenge. CS 231N 7, 7 (2015), 3.

[27]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998), 2278–2324.

[28]

Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. 2020. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations.

[29]

Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. 2021. Contrastive clustering. In Proceedings of the 2021 AAAI Conference on Artificial Intelligence (AAAI’21).

[30]

Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, and Xi Peng. 2022. Twin contrastive learning for online clustering. International Journal of Computer Vision 130, 9 (2022), 2205–2221.

Digital Library

[31]

Zhihui Li, Feiping Nie, Xiaojun Chang, Liqiang Nie, Huaxiang Zhang, and Yi Yang. 2018. Rank-constrained spectral clustering with flexible embedding. IEEE Transactions on Neural Networks and Learning Systems 29, 12 (2018), 6073–6082.

[32]

Zhihui Li, Feiping Nie, Xiaojun Chang, Yi Yang, Chengqi Zhang, and Nicu Sebe. 2018. Dynamic affinity graph construction for spectral clustering using multiple features. IEEE Transactions on Neural Networks and Learning Systems 29, 12 (2018), 6323–6332.

[33]

Juncheng Lv, Zhao Kang, Xiao Lu, and Zenglin Xu. 2021. Pseudo-supervised deep subspace clustering. IEEE Transactions on Image Processing 30 (2021), 5252–5263.

[34]

J. MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics. University of California Press, 281–297.

[35]

Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. 2018. A survey of clustering with deep learning: From the perspective of network architecture. In IEEE Access 6 (2018), 39501–39514.

[36]

Gregory Murphy. 2004. The Big Book of Concepts. MIT press.

[37]

Chuang Niu and Ge Wang. 2021. Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions on Image Processing 31 (2021), 7264–7278.

[38]

Chuang Niu, Jun Zhang, Ge Wang, and Jimin Liang. 2020. Gatcluster: Self-supervised gaussian-attention network for image clustering. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16. Springer, 735–751.

Digital Library

[39]

Foivos Ntelemis, Yaochu Jin, and Spencer A. Thomas. 2022. Information maximization clustering via multi-view self-labelling. Knowledge-Based Systems 250 (2022), 109042.

[40]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv:1807.03748.

[41]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. Advances in Neural Information Processing Systems.

[42]

Xi Peng, Jiashi Feng, Shijie Xiao, Wei-Yun Yau, Joey Tianyi Zhou, and Songfan Yang. 2018. Structured autoencoders for subspace clustering. IEEE Transactions on Image Processing 27, 10 (2018), 5076–5086.

[43]

Yazhou Ren, Jingyu Pu, Zhimeng Yang, Jie Xu, Guofeng Li, Xiaorong Pu, Philip S. Yu, and Lifang He. 2022. Deep clustering: A comprehensive survey. arXiv:2210.04142. Retrieved from https://arxiv.org/abs/2210.04142

[44]

Mohammadreza Sadeghi, Hadi Hojjati, and Narges Armanfard. 2022. C3: Cross-instance guided contrastive clustering. arXiv: 2211.07136.

[45]

Amit Saxena, Mukesh Prasad, Akshansh Gupta, Neha Bharill, Om Prakash Patel, Aruna Tiwari, Meng Joo Er, Weiping Ding, and Chin-Teng Lin. 2017. A review of clustering techniques and developments. Neurocomputing 267 (2017), 664–681.

[46]

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision. 618–626.

[47]

Yaling Tao, Kentaro Takagi, and Kouta Nakata. 2021. Clustering-friendly representation learning via instance discrimination and feature decorrelation. In International Conference on Learning Representations.

[48]

Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive multiview coding. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. Springer, 776–794.

Digital Library

[49]

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. 2020. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision. Springer, 268–285.

Digital Library

[50]

Tuo Wang, Xiang Zhang, Long Lan, and Zhigang Luo. 2022. Local-to-global deep clustering on approximate uniform manifold. IEEE Transactions on Knowledge and Data Engineering 35, 5 (2022), 5035–5046.

[51]

Yu Wang, Chuan Chen, Jinrong Lai, Lele Fu, Yuren Zhou, and Zibin Zheng. 2022. A self-representation method with local similarity preserving for fast multi-view outlier detection. ACM Transactions on Knowledge Discovery from Data 17, 1 (2022), 1–20.

[52]

Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. 2019. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8150–8159.

[53]

Lirong Wu, Zicheng Liu, Zelin Zang, Jun Xia, Siyuan Li, Stan Li, et al. 2020. Deep clustering and representation learning that preserves geometric structures. In International Conference on Learning Representations.

[54]

Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In Proceedings of the International Conference on Machine Learning. PMLR, 478–487.

[55]

Chaoyang Xu, Renjie Lin, Jinyu Cai, and Shiping Wang. 2022. Deep image clustering by fusing contrastive learning and neighbor relation mining. Knowledge-Based Systems 238 (2022), 107967.

[56]

Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. 2022. Multi-level feature learning for contrastive multi-view clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16051–16060.

[57]

Caixia Yan, Xiaojun Chang, Minnan Luo, Qinghua Zheng, Xiaoqin Zhang, Zhihui Li, and Feiping Nie. 2020. Self-weighted robust LDA for multiclass classification with edge classes. ACM Transactions on Intelligent Systems and Technology (TIST) 12, 1 (2020), 1–19.

Digital Library

[58]

Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. 2017. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In Proceedings of the International Conference on Machine Learning. PMLR, 3861–3870.

[59]

Jianwei Yang, Devi Parikh, and Dhruv Batra. 2016. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5147–5156.

[60]

Linxiao Yang, Ngai-Man Cheung, Jiaying Li, and Jun Fang. 2019. Deep clustering by gaussian mixture variational autoencoders with graph embedding. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6440–6449.

[61]

Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. 2020. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6688–6697.

[62]

Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. 2020. Deep robust clustering by contrastive learning. arXiv:2008.03030

[63]

Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, and Xian Sheng Hua. 2021. Graph contrastive clustering. Proceedings of the IEEE/CVF International Conference on Computer Vision. 9224–9233.

Cited By

Li HZhang LSu KYu W(2024)MICCF: A Mutual Information Constrained Clustering Framework for Learning Clustering-Oriented Feature RepresentationsACM Transactions on Knowledge Discovery from Data10.1145/367240218:8(1-22)Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3672402

Index Terms

Structure-Driven Representation Learning for Deep Clustering
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis
    2. Machine learning approaches
      1. Learning latent representations

Recommendations

Centroids-guided deep multi-view K-means clustering
Abstract
With the progress of deep learning used in unsupervised learning, deep approach based multi-view clustering methods have been increasingly proposed in recent years. However, in most of these methods, deep representation learning is not ...
Deep boundary-aware clustering by jointly optimizing unsupervised representation learning
Abstract
Deep clustering obtains feature representation generally and then performs clustering for high dimension real-world data. However, conventional solutions are two-stage embedding learning-based methods and these two processes are separate and ...
Point Symmetry-based Deep Clustering
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

Clustering is a central task in unsupervised learning. Recent advances that perform clustering into learned deep features (such as DEC[14], IDEC [6] or VaDe [10]) have shown improvements over classical algorithms, but most of them are based on the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data

ACM Transactions on Knowledge Discovery from Data Volume 18, Issue 1

January 2024

854 pages

ISSN:1556-4681

EISSN:1556-472X

DOI:10.1145/3613504

Editor:
Charu Aggarwal
IBM T. J. Watson Research, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2023

Online AM: 08 September 2023

Accepted: 30 August 2023

Revised: 21 June 2023

Received: 24 September 2022

Published in TKDD Volume 18, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Key Research and Development Program
Joint Foundation of the Ministry of Education
Beijing Natural Science Foundation
Fundamental Research Funds for the Central Universities
Chinese Academy of Sciences

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
429
Total Downloads

Downloads (Last 12 months)429
Downloads (Last 6 weeks)27

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li HZhang LSu KYu W(2024)MICCF: A Mutual Information Constrained Clustering Framework for Learning Clustering-Oriented Feature RepresentationsACM Transactions on Knowledge Discovery from Data10.1145/367240218:8(1-22)Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3672402

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents