research-article

Tailoring density estimation via reproducing kernel moment matching

Authors:

Arthur Gretton,

Bernhard SchölkopfAuthors Info & Claims

ICML '08: Proceedings of the 25th international conference on Machine learning

Pages 992 - 999

https://doi.org/10.1145/1390156.1390281

Published: 05 July 2008 Publication History

Abstract

Moment matching is a popular means of parametric density estimation. We extend this technique to nonparametric estimation of mixture models. Our approach works by embedding distributions into a reproducing kernel Hilbert space, and performing moment matching in that space. This allows us to tailor density estimators to a function class of interest (i.e., for which we would like to compute expectations). We show our density estimation approach is useful in applications such as message compression in graphical models, and image classification and retrieval.

References

[1]

Altun, Y., & Smola, A. (2006). Unifying divergence minimization and statistical inference via convex duality. In COLT 2006.

Digital Library

[2]

Barndorff-Nielsen, O. E. (1978). Information and Exponential Families in Statistical Theory.

[3]

Bartlett, P. L., & Mendelson, S. (2002). Rademacher and Gaussian complexities: Risk bounds and structural results. JMLR, 3, 463--482.

Digital Library

[4]

de Farias, N., & Roy, B. (2004). On constraint sampling in the linear programming approach to approximate dynamic programming. Math. Oper. Res., 29(3).

Digital Library

[5]

Doucet, A., de Freitas, N., & Gordon, N. (2001). Sequential Monte Carlo Methods in Practice. Springer-Verlag.

[6]

Dudík, M., Phillips, S., & Schapire, R. (2004). Performance guarantees for regularized maximum entropy density estimation. In COLT 2004.

[7]

DuMouchel, W., Volinsky, C., Cortes, C., Pregibon, D., & Johnson, T. (1999). Squashing flat files flatter. In KDD 1999.

Digital Library

[8]

Girolami, M., & He, C. (2003). Probability density estimation from optimally condensed data samples. IEEE TPAMI, 25(10), 1253--1264.

Digital Library

[9]

Greenspan, H., Dvir, G., & Rubner, Y. (2002). Context-based image modeling. In ICPR 2002.

Digital Library

[10]

Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., & Smola, A. (2007). A kernel method for the two-sample-problem. In NIPS 2007.

Digital Library

[11]

Jebara, T., Kondor, R., & Howard, A. (2004). Probability product kernels. JMLR, 5, 819--844.

Digital Library

[12]

McLachlan, G., & Basford, K. (1988). Mixture Models: Inference and Applications to Clustering. Marcel Dekker.

[13]

Minka, T. (2001). Expectation Propagation for approximate Bayesian inference. Ph.D. thesis, MIT.

[14]

Parzen, E. (1962). On estimation of a probability density function and mode. Ann. Math. Stat., 33, 1065--1076.

[15]

Rubner, Y., Tomasi, C., & Guibas, L. (2000). The earth mover's distance as a metric for image retrieval. Intl. J. Computer Vision, 40(2), 99--121.

Digital Library

[16]

Shawe-Taylor, J., & Dolia, A. (2007). A framework for probability density estimation. In AISTATS 2007.

[17]

Silverman, B. W. (1986). Density estimation for statistical and data analysis. Monographs on statistics and applied probability. Chapman and Hall.

[18]

Smola, A., Gretton, A., Song, L., & Schöölkopf, B. (2007). A Hilbert space embedding for distributions. In ALT 2007.

Digital Library

[19]

Steinwart, I. (2002). The influence of the kernel on the consistency of support vector machines. JMLR, 2, 463--482.

Digital Library

[20]

Vapnik, V., & Mukherjee, S. (2000). Support vector method for multivariate density estimation. In NIPS 12.

[21]

Wainwright, M. J., & Jordan, M. I. (2003). Graphical models, exponential families, and variational inference. Tech. Rep. 649, UC Berkeley.

Cited By

Lakshmanan RPichler A(2024)Unbalanced Optimal Transport and Maximum Mean Discrepancies: Interconnections and Rapid EvaluationJournal of Scientific Computing10.1007/s10915-024-02586-2100:3Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s10915-024-02586-2
Wu GLindquist A(2024)A non-classical parameterization for density estimation using sample momentsStatistical Papers10.1007/s00362-024-01563-zOnline publication date: 20-May-2024
https://doi.org/10.1007/s00362-024-01563-z
Oates C(2024)Minimum Kernel Discrepancy EstimatorsMonte Carlo and Quasi-Monte Carlo Methods10.1007/978-3-031-59762-6_6(133-161)Online publication date: 13-Jul-2024
https://doi.org/10.1007/978-3-031-59762-6_6
Show More Cited By

Index Terms

Tailoring density estimation via reproducing kernel moment matching
1. Computing methodologies
  1. Machine learning
  2. Modeling and simulation
    1. Model development and analysis
      1. Modeling methodologies
2. Mathematics of computing
  1. Probability and statistics
    1. Distribution functions
    2. Nonparametric statistics

Recommendations

Mixture density estimation with group membership functions

The mixture density model has been extensively studied in the field of statistical pattern recognition. And the EM algorithm has been well known as a convenient and efficient tool to iteratively compute the maximum likelihood estimates of mixture model ...
Robust kernel density estimation

We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical M-estimation. ...
Kernel mixture model for probability density estimation in Bayesian classifiers

Estimating reliable class-conditional probability is the prerequisite to implement Bayesian classifiers, and how to estimate the probability density functions (PDFs) is also a fundamental problem for other probabilistic induction algorithms. The finite ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '08: Proceedings of the 25th international conference on Machine learning

July 2008

1310 pages

ISBN:9781605582054

DOI:10.1145/1390156

General Chair:
William Cohen
Carnegie Mellon University
,
Program Chairs:
Andrew McCallum
University of Massachusetts Amherst
,
Sam Roweis
University of Toronto and Google

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Pascal
University of Helsinki
Xerox
Federation of Finnish Learned Societies
Google Inc.
NSF
Machine Learning Journal/Springer
Microsoft Research: Microsoft Research
Intel: Intel
Yahoo!
Helsinki Institute for Information Technology
IBM: IBM

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Seventh Framework Programme

Conference

ICML '08

Sponsor:

Microsoft Research
Intel
IBM

ICML '08: The 25th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

July 5 - 9, 2008

Helsinki, Finland

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
310
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)1

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lakshmanan RPichler A(2024)Unbalanced Optimal Transport and Maximum Mean Discrepancies: Interconnections and Rapid EvaluationJournal of Scientific Computing10.1007/s10915-024-02586-2100:3Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s10915-024-02586-2
Wu GLindquist A(2024)A non-classical parameterization for density estimation using sample momentsStatistical Papers10.1007/s00362-024-01563-zOnline publication date: 20-May-2024
https://doi.org/10.1007/s00362-024-01563-z
Oates C(2024)Minimum Kernel Discrepancy EstimatorsMonte Carlo and Quasi-Monte Carlo Methods10.1007/978-3-031-59762-6_6(133-161)Online publication date: 13-Jul-2024
https://doi.org/10.1007/978-3-031-59762-6_6
Yoon SPark FYun GKim INoh YOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Variational weighting for kernel density ratiosProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666344(5010-5027)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666344
Küppers FSchneider JHaselhoff A(2023)Parametric and Multivariate Uncertainty Calibration for Regression and Object DetectionComputer Vision – ECCV 2022 Workshops10.1007/978-3-031-25072-9_30(426-442)Online publication date: 18-Feb-2023
https://doi.org/10.1007/978-3-031-25072-9_30
Brodu NCrutchfield J(2022) Discovering causal structure with reproducing-kernel Hilbert space ε Chaos: An Interdisciplinary Journal of Nonlinear Science10.1063/5.006282932:2(023103)Online publication date: Feb-2022
https://doi.org/10.1063/5.0062829
Kaya UGopireddy SUrbanetz NNopens IVerwaeren J(2022)Predicting the hydrodynamic properties of a bioreactor: Conditional density estimation as a surrogate model for CFD simulationsChemical Engineering Research and Design10.1016/j.cherd.2022.03.042182(342-359)Online publication date: Jun-2022
https://doi.org/10.1016/j.cherd.2022.03.042
Van Hauwermeiren DStock MDe Beer TNopens I(2020)Predicting Pharmaceutical Particle Size Distributions Using Kernel Mean EmbeddingPharmaceutics10.3390/pharmaceutics1203027112:3(271)Online publication date: 16-Mar-2020
https://doi.org/10.3390/pharmaceutics12030271
Yao LDimitrakopoulos RGamache M(2019)High-Order Sequential Simulation via Statistical Learning in Reproducing Kernel Hilbert SpaceMathematical Geosciences10.1007/s11004-019-09843-3Online publication date: 7-Dec-2019
https://doi.org/10.1007/s11004-019-09843-3
Phillips JTai W(2019)Near-Optimal Coresets of Kernel Density EstimatesDiscrete & Computational Geometry10.1007/s00454-019-00134-6Online publication date: 25-Sep-2019
https://doi.org/10.1007/s00454-019-00134-6
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents