article

Bayesian parameter estimation via variational methods

Authors:

Tommi S. Jaakkola,

Michael I. JordanAuthors Info & Claims

Statistics and Computing, Volume 10, Issue 1

Pages 25 - 37

https://doi.org/10.1023/A:1008932416310

Published: 01 January 2000 Publication History

Abstract

We consider a logistic regression model with a Gaussian prior distribution over the parameters. We show that an accurate variational transformation can be used to obtain a closed form approximation to the posterior distribution of the parameters thereby yielding an approximate posterior predictive model. This approach is readily extended to binary graphical model with complete observations. For graphical models with incomplete observations we utilize an additional variational transformation and again obtain a closed form approximation to the posterior. Finally, we show that the dual of the regression problem gives a latent variable density model, the variational formulation of which leads to exactly solvable EM updates.

References

[1]

Bernardo J. and Smith A. 1994. Bayesian Theory. New York, Wiley.

[2]

Everitt B. 1984. An Introduction to Latent Variable Models. Cambridge University Press.

[3]

Gelman A. 1995. Bayesian Data Analysis. Boca Raton, FL, CRC Press.

[4]

Gilks W., Richardson, S., and Spiegelhalter D. (1996). Markov Chain Monte Carlo in Practice. London, Chapman and Hall.

[5]

Heckerman D., Geiger D., and Chickering D. 1995. Learning Bayesian networks: the combination of knowledge and statistical data. Machine Learning 20: 197-244.

[6]

Hinton G. and van Camp D. 1993. Keeping neural networks simple by minimizing the description length of the weights. In: Proceedings of the 6th Annual Workshop on Computational Learning Theory. New York, ACM Press.

Digital Library

[7]

Jordan M., Ghahramani Z., Jaakkola T., and Saul L. 1999. An introduction to variational methods in graphical models. In: Jordan M.I. (Ed.), Learning in Graphical Models. Cambridge, MA, MIT Press.

Digital Library

[8]

MacKay D. 1997. Ensemble learning for hidden Markov models. Unpublished manuscript. Department of Physics, University of Cambridge. Available on the web at http://wol.ra.phy. cam.ac.uk/mackay.

[9]

McCullagh P. and Nelder J. 1983. Generalized Linear Models. London, Chapman and Hall.

[10]

Nadal J-P. and Parga N. 1994. Duality between learning machines: A bridge between supervised and unsupervised learning. Neural Computation 6(3): 491-508.

Digital Library

[11]

Neal R. 1992. Connectionist learning of belief networks. Artificial Intelligence 56: 71-113.

Digital Library

[12]

Neal R. 1993. Probabilistic inference using Markov chain Monte Carlo methods. Technical report CRG-TR-93-1, University of Toronto.

[13]

Parisi G. 1988. Statistical field theory. Redwood City, CA, Addison-Wesley.

[14]

Rockafellar R. 1972. Convex Analysis. Princeton University Press.

[15]

Rustagi J. 1976. Variational Methods in Statistics. New York, Academic Press.

[16]

Saul L., Jaakkola T., and Jordan M. 1996. Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research 4: 61- 76.

[17]

Saul L. and Jordan M. 1996. Exploiting tractable substructures in intractable networks. In: Touretzky D.S., Mozer M.C., and Hasselmo M.E. (Eds.), Advances in Neural Information Processing Systems 8. Cambridge MA, MIT Press.

[18]

Spiegelhalter D. and Lauritzen S. 1990. Sequential updating of conditional probabilities on directed graphical structures. Networks 20: 579-605.

[19]

Thomas A., Spiegelhalter D., and GilksW. 1992. BUGS: A program to perform Bayesian inference using Gibbs sampling. In: Bayesian Statistics 4. Clarendon Press.

[20]

Tipping M. 1999. Probabilistic visualisation of high-dimensional binary data. Advances in Neural Information Processing Systems 11.

Digital Library

Cited By

Komodromos MEvangelou MFilippi SSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Logistic variational bayes revisitedProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693074(25087-25104)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693074
Liu ZLi ZWang JHe YWooldridge MDy JNatarajan S(2024)Full Bayesian significance testing for neural networksProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i8.28731(8841-8849)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i8.28731
Liu JWu XXie YTang ZXie YGong S(2024)Small samples-oriented intrinsically explainable machine learning using Variational Bayesian Logistic RegressionExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121138235:COnline publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121138
Show More Cited By

Bayesian parameter estimation via variational methods
1. Computing methodologies
2. Mathematics of computing
  1. Mathematical analysis
    1. Functional analysis
  2. Probability and statistics
    1. Probabilistic reasoning algorithms

Recommendations

An Introduction to Variational Methods for Graphical Models

This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields). We present a number of examples of graphical models, including the QMR-DT database, ...
Parameter expanded variational Bayesian methods
NIPS'06: Proceedings of the 20th International Conference on Neural Information Processing Systems

Bayesian inference has become increasingly important in statistical machine learning. Exact Bayesian calculations are often not feasible in practice, however. A number of approximate Bayesian methods have been proposed to make such calculations practical,...
Bayesian Estimation of Beta Mixture Models with Variational Inference

Bayesian estimation of the parameters in beta mixture models (BMM) is analytically intractable. The numerical solutions to simulate the posterior distribution are available, but incur high computational cost. In this paper, we introduce an approximation ...

Comments

Information & Contributors

Information

Published In

cover image Statistics and Computing

Statistics and Computing Volume 10, Issue 1

January 2000

80 pages

ISSN:0960-3174

Issue’s Table of Contents

Copyright © Copyright © 2000 Kluwer Academic Publishers.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 January 2000

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

104
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Komodromos MEvangelou MFilippi SSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Logistic variational bayes revisitedProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693074(25087-25104)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693074
Liu ZLi ZWang JHe YWooldridge MDy JNatarajan S(2024)Full Bayesian significance testing for neural networksProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i8.28731(8841-8849)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i8.28731
Liu JWu XXie YTang ZXie YGong S(2024)Small samples-oriented intrinsically explainable machine learning using Variational Bayesian Logistic RegressionExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121138235:COnline publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121138
Benedetti LBoniardi EChiani LGhirri JMastropietro MCappozzo ADenti F(2024)Variational inference for semiparametric Bayesian novelty detection in large datasetsAdvances in Data Analysis and Classification10.1007/s11634-023-00569-z18:3(681-703)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s11634-023-00569-z
Xia YChen JJiang D(2024)Variational Bayesian analysis for two-part latent variable modelComputational Statistics10.1007/s00180-023-01417-639:4(2259-2290)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s00180-023-01417-6
Sertkaya ERyzhov I(2023)Approximate Gaussian Process Regression with Pairwise Comparison DataProceedings of the Winter Simulation Conference10.5555/3643142.3643434(3496-3507)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3643142.3643434
Aktukmak MZhu HChevrette MNepper JMagesh SHandelsman JHero A(2023)A Graphical Model for Fusing Diverse Microbiome DataIEEE Transactions on Signal Processing10.1109/TSP.2023.330946471(3399-3412)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TSP.2023.3309464
Wang KLi AWang XSun L(2023)Study on Credit Risk Control by Variational InferenceWeb Information Systems Engineering – WISE 202310.1007/978-981-99-7254-8_62(801-809)Online publication date: 25-Oct-2023
https://dl.acm.org/doi/10.1007/978-981-99-7254-8_62
Karlsson RWong DKawabata KThompson SSakai N(2022)Probabilistic Rainfall Estimation from Automotive Lidar2022 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV51971.2022.9827119(37-44)Online publication date: 4-Jun-2022
https://dl.acm.org/doi/10.1109/IV51971.2022.9827119
Lee YJo SLee J(2022)A variational inference for the Lévy adaptive regression with multiple kernelsComputational Statistics10.1007/s00180-022-01200-z37:5(2493-2515)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s00180-022-01200-z
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents