research-article

On Weighting Clustering

Authors:

Frank NielsenAuthors Info & Claims

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 28, Issue 8

Pages 1223 - 1235

https://doi.org/10.1109/TPAMI.2006.168

Published: 01 August 2006 Publication History

Abstract

Recent papers and patents in iterative unsupervised learning have emphasized a new trend in clustering. It basically consists of penalizing solutions via weights on the instance points, somehow making clustering move toward the hardest points to cluster. The motivations come principally from an analogy with powerful supervised classification methods known as boosting algorithms. However, interest in this analogy has so far been mainly borne out from experimental studies only. This paper is, to the best of our knowledge, the first attempt at its formalization. More precisely, we handle clustering as a constrained minimization of a Bregman divergence. Weight modifications rely on the local variations of the expected complete log-likelihoods. Theoretical results show benefits resembling those of boosting algorithms and bring modified (weighted) versions of clustering algorithms such as k\hbox{-}\rm means, fuzzy c\hbox{-}\rm means, Expectation Maximization (EM), and k\hbox{-}\rm harmonic means. Experiments are provided for all these algorithms, with a readily available code. They display the advantages that subtle data reweighting may bring to clustering.

References

[1]

C. Gentile and M. Warmuth, “Proving Relative Loss Bounds for On-Line Learning Algorithms Using Bregman Divergences,” Proc. Tutorials 13th Int'l Conf. Computational Learning Theory, 2000.

[2]

J. Computer and System Sciences, vol. 55, pp. 119-139, 1997.

[3]

J. Kivinen and M. Warmuth, “Boosting as Entropy Projection,” Proc. 12th Ann. Conf. Computational Learning Theory, pp. 134-144, 1999.

Digital Library

[4]

R.E. Schapire and Y. Singer, “Improved Boosting Algorithms Using Confidence-Rated Predictions,” Proc. 11th Int'l Conf. Computational Learning Theory, pp. 80-91, 1998.

Digital Library

[5]

M.J. Kearns, “Thoughts on Hypothesis Boosting,” ML class project, 1988.

[6]

A. Banerjee, S. Merugu, I. Dhillon, and J. Ghosh, “Clustering with Bregman Divergences,” Proc. Fourth SIAM Int'l Conf. Data Mining, pp. 234-245, 2004.

[7]

A. Banerjee, S. Merugu, I. Dhillon, and J. Ghosh, “Clustering with Bregman Divergences,” J. Machine Learning Research, vol. 6, pp. 1705-1749, 2005.

Digital Library

[8]

G. Hammerly and C. Elkan, “Alternatives to the $k \hbox{-}\rm Means$ Algorithm that Find Better Clusterings,” Proc. 11th ACM Int'l Conf. Information and Knowledge Management, pp. 600-607, 2002.

Digital Library

[9]

B. Zhang, “Generalized $k \hbox{-}\rm Harmonic$ Means,” Technical Report TR-HPL-2000-137, Hewlett Packard Labs, 2000.

[10]

A. Topchy, B. Minaei-Bidgoli, A.-K. Jain, and W.-F. Punch, “Adaptive Clustering Ensembles,” Proc. 17th Int'l Conf. Pattern Recognition, pp. 272-275, 2004.

Digital Library

[11]

B. Zhang, M. Hsu, and U. Dayal, “$k \hbox{-}\rm Harmonic$ Means— A Spatial Clustering Algorithm with Boosting,” Temporal, Spatial, and Spatio-Temporal Data Mining, pp. 31-45, 2000.

Digital Library

[12]

B. Zhang, M. Hsu, and U. Dayal, “Harmonic Average Based Clustering Method and System,” US Patent 6,584,433, 2000.

[13]

J. McQueen, “Some Methods for Classification and Analysis of Multivariate Observations,” Proc. Fifth Berkeley Symp. Math. Statistics and Probability, pp. 281-297, 1967.

[14]

J.-C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, 1981.

[15]

A.P. Dempster, N.M. Laird, and D.B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. Royal Statistical Soc. B, vol. 39, pp. 1-38, 1977.

[16]

M.J. Kearns, Y. Mansour, and A.Y. Ng, “An Information-Theoretic Analysis of Hard and Soft Assignment Methods for Clustering,” Proc. 13th Int'l Conf. Uncertainty in Artificial Intelligence, pp. 282-293, 1997.

Digital Library

[17]

I. Budimir, S. Dragomir, and J. Pecaric, “Further Reverse Results for Jensen's Discrete Inequality and Applications in Information Theory,” J. Inequalities in Pure and Applied Math., vol. 3, 2000.

[18]

L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. Springer, 1996.

[19]

H. Attias, “A Variational Bayesian Framework for Graphical Models,” Advances in Neural Information Processing Systems 12, pp. 209-215, 1999.

[20]

M.-J. Beal and Z. Ghahramani, “The Variational Bayesian EM Algorithm for Incomplete Data: With Application to Scoring Graphical Models,” Bayesian Statistics, vol. 7, pp. 453-464, 2003.

Cited By

Yang L(2022)Research on Precision Marketing Strategy of Ski Tourism Market Based on Clustering AlgorithmProceedings of the 3rd Asia-Pacific Conference on Image Processing, Electronics and Computers10.1145/3544109.3544156(253-258)Online publication date: 14-Apr-2022
https://dl.acm.org/doi/10.1145/3544109.3544156
Anand SKumar S(2022)Experimental Comparisons of Clustering Approaches for Data RepresentationACM Computing Surveys10.1145/349038455:3(1-33)Online publication date: 30-Mar-2022
https://dl.acm.org/doi/10.1145/3490384
Rani SLakhwani KKumar S(2022)Three dimensional objects recognition & pattern recognition technique; related challenges: A reviewMultimedia Tools and Applications10.1007/s11042-022-12412-281:12(17303-17346)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1007/s11042-022-12412-2
Show More Cited By

Index Terms

On Weighting Clustering
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
      1. Probabilistic reasoning
      2. Vagueness and fuzzy logic
  2. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis

Recommendations

Bregman bubble clustering: A robust framework for mining dense clusters

In classical clustering, each data point is assigned to at least one cluster. However, in many applications only a small subset of the available data is relevant for the problem and the rest needs to be ignored in order to obtain good clusters. Certain ...
Multitask Bregman clustering

Traditional clustering methods deal with a single clustering task on a single data set. In some newly emerging applications, multiple similar clustering tasks are involved simultaneously. In this case, we not only desire a partition for each task, but ...
Hybridization of Expectation-Maximization and K-Means Algorithms for Better Clustering Performance

Abstract The present work proposes hybridization of Expectation-Maximization EM and K-means techniques as an attempt to speed-up the clustering process. Even though both the K-means and EM techniques look into different areas, K-means can be viewed as ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Pattern Analysis and Machine Intelligence

IEEE Transactions on Pattern Analysis and Machine Intelligence Volume 28, Issue 8

August 2006

172 pages

ISSN:0162-8828

Issue’s Table of Contents

Copyright © 2006.

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 August 2006

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang L(2022)Research on Precision Marketing Strategy of Ski Tourism Market Based on Clustering AlgorithmProceedings of the 3rd Asia-Pacific Conference on Image Processing, Electronics and Computers10.1145/3544109.3544156(253-258)Online publication date: 14-Apr-2022
https://dl.acm.org/doi/10.1145/3544109.3544156
Anand SKumar S(2022)Experimental Comparisons of Clustering Approaches for Data RepresentationACM Computing Surveys10.1145/349038455:3(1-33)Online publication date: 30-Mar-2022
https://dl.acm.org/doi/10.1145/3490384
Rani SLakhwani KKumar S(2022)Three dimensional objects recognition & pattern recognition technique; related challenges: A reviewMultimedia Tools and Applications10.1007/s11042-022-12412-281:12(17303-17346)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1007/s11042-022-12412-2
Suryanarayana GPrakash K LMahesh PBhaskar T(2022)Novel dynamic k-modes clustering of categorical and non categorical dataset with optimized genetic algorithm based feature selectionMultimedia Tools and Applications10.1007/s11042-022-12126-581:17(24399-24418)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s11042-022-12126-5
Narayana GKolli K(2021)Fuzzy K-means clustering with fast density peak clustering on multivariate kernel estimator with evolutionary multimodal optimization clusters on a large datasetMultimedia Tools and Applications10.1007/s11042-020-09718-480:3(4769-4787)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s11042-020-09718-4
Gupta YSaini A(2019)A new swarm-based efficient data clustering approach using KHM and fuzzy logicSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-018-3514-123:1(145-162)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.1007/s00500-018-3514-1
Ke SLin WTsai CHu Y(2017)Soft estimation by hierarchical classification and regressionNeurocomputing10.1016/j.neucom.2016.12.037234:C(27-37)Online publication date: 19-Apr-2017
https://dl.acm.org/doi/10.1016/j.neucom.2016.12.037
Kumar KReddy A(2017)An efficient k-means clustering filtering algorithm using density based initial cluster centersInformation Sciences: an International Journal10.1016/j.ins.2017.07.036418:C(286-301)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1016/j.ins.2017.07.036
Tencer LReznakova MCheriet M(2017)UFuzzyApplied Soft Computing10.1016/j.asoc.2017.05.01859:C(1-18)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1016/j.asoc.2017.05.018
Huang WOh SPedrycz W(2017)Hybrid fuzzy polynomial neural networks with the aid of weighted fuzzy clustering method and fuzzy polynomial neuronsApplied Intelligence10.1007/s10489-016-0844-546:2(487-508)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s10489-016-0844-5
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents