research-article

Data spectroscopy: learning mixture models using eigenspaces of convolution operators

Authors:

Tao Shi,

Mikhail Belkin,

Bin YuAuthors Info & Claims

ICML '08: Proceedings of the 25th international conference on Machine learning

Pages 936 - 943

https://doi.org/10.1145/1390156.1390274

Published: 05 July 2008 Publication History

Get Access

Abstract

In this paper we develop a spectral framework for estimating mixture distributions, specifically Gaussian mixture models. In physics, spectroscopy is often used for the identification of substances through their spectrum. Treating a kernel function K(x, y) as "light" and the sampled data as "substance", the spectrum of their interaction (eigenvalues and eigenvectors of the kernel matrix K) unveils certain aspects of the underlying parametric distribution p, such as the parameters of a Gaussian mixture. Our approach extends the intuitions and analyses underlying the existing spectral techniques, such as spectral clustering and Kernel Principal Components Analysis (KPCA).

We construct algorithms to estimate parameters of Gaussian mixture models, including the number of mixture components, their means and covariance matrices, which are important in many practical applications. We provide a theoretical framework and show encouraging experimental results.

References

[1]

M. Belkin, P. Niyogi, Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering, NIPS 2001.

Google Scholar

[2]

S. Dasgupta, Learning mixtures of Gaussians, FOCS 1999.

Digital Library

Google Scholar

[3]

A. Dempster, N. Laird, D. Rubin, Maximum-likelihood from incomplete data via the em algorithm, Journal of Royal Statistics Society, Ser. B, 39 (1997), pp. 1--38.

Google Scholar

[4]

R. Kannan, H. Salmasian, S. Vempala, The spectral method for general mixture models, COLT 2005.

Digital Library

Google Scholar

[5]

T. Kato, Perturbation Theory for Linear Operators, Springer-Verlag, 1966.

Google Scholar

[6]

V. Koltchinskii and E. Giné, Random matrix approximation of spectra of integral operators, Bernoulli, 6 (2000), pp. 113--167.

Crossref

Google Scholar

[7]

S. MacEachern and P. Muller, Estimating mixture of Dirichlet process models, Journal of Computational and Graphical Statistics, 7 (1998), pp. 223--238.

Google Scholar

[8]

A. Ng, M. Jordan, and Y. Weiss, On spectral clustering: Analysis and an algorithm, NIPS 2001.

Digital Library

Google Scholar

[9]

K. Pearson, Contributions to the mathematical theory of evolution, Phil. Trans. Royal Soc., 185A, (1894), pp. 71--110.

Crossref

Google Scholar

[10]

R. Redner and H. Walker, Mixture densities, maximum likelihood and the em algorithm, SIAM Review, 26 (1984), pp. 195--239.

Digital Library

Google Scholar

[11]

B. Schölkopf, A. J. Smola, and K.-R. Müller, Kernel principal component analysis, Advances in kernel methods: support vector learning, (1999), pp. 327--352.

Digital Library

Google Scholar

[12]

C. Williams, M. Seeger, The effect of the input density distribution on kernel-based classifiers, ICML2000.

Digital Library

Google Scholar

[13]

H. Zhu, C. Williams, R. Rohwer, and M. Morcinie, Gaussian regression and optimal finite dimensional linear models, in Neural networks and machine learning, C. Bishop, ed., 1998.

Google Scholar

Cited By

View all

Zach IDvorkind TTalmon R(2024)Graph signal interpolation and extrapolation over manifold of Gaussian mixtureSignal Processing10.1016/j.sigpro.2023.109308216(109308)Online publication date: Mar-2024
https://doi.org/10.1016/j.sigpro.2023.109308
Le TClarke B(2021)Interpreting uninterpretable predictors: kernel methods, Shtarkov solutions, and random forestsStatistical Theory and Related Fields10.1080/24754269.2021.1974157(1-19)Online publication date: 8-Sep-2021
https://doi.org/10.1080/24754269.2021.1974157
Zhou Y(2021)Range Shifts Under Constant-Speed and Accelerated Climate WarmingBulletin of Mathematical Biology10.1007/s11538-021-00963-884:1Online publication date: 17-Nov-2021
https://doi.org/10.1007/s11538-021-00963-8
Show More Cited By

Index Terms

Data spectroscopy: learning mixture models using eigenspaces of convolution operators
1. Computing methodologies
  1. Machine learning
  2. Modeling and simulation
    1. Model development and analysis
      1. Model verification and validation
2. Mathematics of computing
  1. Mathematical analysis
    1. Differential equations
      1. Partial differential equations
  2. Probability and statistics
    1. Statistical paradigms
      1. Statistical graphics

Recommendations

Nonnegative Sparse Blind Source Separation for NMR Spectroscopy by Data Clustering, Model Reduction, and $\ell_1$ Minimization

Motivated by applications in nuclear magnetic resonance (NMR) spectroscopy, we introduce a novel blind source separation (BSS) approach to treat nonnegative and correlated data. We consider the (over)-determined case where $n$ sources are to be separated ...
A Study of Identifiability for Blind Signal Separation via Nonorthogonal Joint Diagonalization
MACE '12: Proceedings of the 2012 Third International Conference on Mechanic Automation and Control Engineering

The identifiability of blind signal separation(BSS) via nonorthogonal joint diagonalization whose eigen-matrices are based on fourth-order cumulant is addressed in this paper. It is demonstrated that the aim of BSS is to pursuit the matrices essentially ...
Independent Component Analysis of Real Data
ICAPR '09: Proceedings of the 2009 Seventh International Conference on Advances in Pattern Recognition

Independent component analysis (ICA) is a statistical method used to discover hidden factors (sources or features) from a set of measurements or observed data such that the sources are maximally independent. The ICA algorithms are able to separate the ...

Comments

Information & Contributors

Information

Published In

ICML '08: Proceedings of the 25th international conference on Machine learning

July 2008

1310 pages

ISBN:9781605582054

DOI:10.1145/1390156

General Chair:
William Cohen
Carnegie Mellon University
,
Program Chairs:
Andrew McCallum
University of Massachusetts Amherst
,
Sam Roweis
University of Toronto and Google

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

ICML '08

Sponsor:

Microsoft Research
Intel
IBM

ICML '08: The 25th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

July 5 - 9, 2008

Helsinki, Finland

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
213
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zach IDvorkind TTalmon R(2024)Graph signal interpolation and extrapolation over manifold of Gaussian mixtureSignal Processing10.1016/j.sigpro.2023.109308216(109308)Online publication date: Mar-2024
https://doi.org/10.1016/j.sigpro.2023.109308
Le TClarke B(2021)Interpreting uninterpretable predictors: kernel methods, Shtarkov solutions, and random forestsStatistical Theory and Related Fields10.1080/24754269.2021.1974157(1-19)Online publication date: 8-Sep-2021
https://doi.org/10.1080/24754269.2021.1974157
Zhou Y(2021)Range Shifts Under Constant-Speed and Accelerated Climate WarmingBulletin of Mathematical Biology10.1007/s11538-021-00963-884:1Online publication date: 17-Nov-2021
https://doi.org/10.1007/s11538-021-00963-8
Fan XYue YSarkar PWang YDaumé HSingh A(2020)On hyperparameter tuning in general clustering problemsProceedings of the 37th International Conference on Machine Learning10.5555/3524938.3525219(2996-3007)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.5555/3524938.3525219
Singer MKrivobokova TMunk A(2017)Kernel partial least squares for stationary dataThe Journal of Machine Learning Research10.5555/3122009.317686718:1(4447-4487)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.5555/3122009.3176867
Tomozei DMassoulié L(2014)Distributed user Profiling via Spectral MethodsStochastic Systems10.1287/11-SSY0364:1(1-43)Online publication date: Jun-2014
https://doi.org/10.1287/11-SSY036
Potworowski JJakuczun WŁęski SWójcik D(2012)Kernel current source density methodNeural Computation10.1162/NECO_a_0023624:2(541-575)Online publication date: 1-Feb-2012
https://dl.acm.org/doi/10.1162/NECO_a_00236
Matsubara YSakurai YFaloutsos CIwata TYoshikawa MYang QAgarwal DPei J(2012)Fast mining and forecasting of complex time-stamped eventsProceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/2339530.2339577(271-279)Online publication date: 12-Aug-2012
https://dl.acm.org/doi/10.1145/2339530.2339577
Myhre JJenssen R(2012)Mixture weight influence on kernel entropy component analysis and semi-supervised learning using the Lasso2012 IEEE International Workshop on Machine Learning for Signal Processing10.1109/MLSP.2012.6349814(1-6)Online publication date: Sep-2012
https://doi.org/10.1109/MLSP.2012.6349814
Schuetter JShi T(2012)Multiple Sample Data Spectroscopic Clustering of Large Datasets Using Nyström ExtensionJournal of Computational and Graphical Statistics10.1080/10618600.2012.67210421:2(338-360)Online publication date: 14-Jun-2012
https://doi.org/10.1080/10618600.2012.672104
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Nonnegative Sparse Blind Source Separation for NMR Spectroscopy by Data Clustering, Model Reduction, and $\ell_1$ Minimization

A Study of Identifiability for Blind Signal Separation via Nonorthogonal Joint Diagonalization

Independent Component Analysis of Real Data

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Nonnegative Sparse Blind Source Separation for NMR Spectroscopy by Data Clustering, Model Reduction, and $\ell_1$ Minimization

A Study of Identifiability for Blind Signal Separation via Nonorthogonal Joint Diagonalization

Independent Component Analysis of Real Data

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations