Article

Free access

A variational approximation for Bayesian networks with discrete and continuous latent variables

Author:

Kevin P. MurphyAuthors Info & Claims

UAI'99: Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence

Pages 457 - 466

Published: 30 July 1999 Publication History

PDF eReader Publisher Site

Abstract

We show how to use a variational approximation to the logistic function to perform approximate inference in Bayesian networks containing discrete nodes with continuous parents. Essentially, we convert the logistic function to a Gaussian, which facilitates exact inference, and then iteratively adjust the variational parameters to improve the quality of the approximation. We demonstrate experimentally that this approximation is much faster than sampling, but comparable in accuracy. We also introduce a simple new technique for handling evidence, which allows us to handle arbitrary distributionson observed nodes, as well as achieving a significant speedup in networks with discrete variables of large cardinality.

References

[1]

S. Alag and A. Agogino. Inference using message propogation and topology transformation in vector Gaussian continuous networks. In Proc. of the Conf. on Uncertainty in AI, 1996.

Digital Library

[2]

A. Becker and D. Geiger. A sufficiently fast algorithm for finding close to optimal junction trees. In Proc. of the Conf. on Uncertainty in AI, 1996.

Digital Library

[3]

J. Binder, D. Koller, S. J. Russell, and K. Kanazawa. Adaptive probabilistic networks with hidden variables Machine Learning, 29:213-244, 1997.

Digital Library

[4]

Y. Bar-Shalom and X. Li. Estimation and Tracking: Principles, Techniques and Software. Artech House, 1993.

[5]

K. C. Chang and R. M. Fung. Symbolic probabalistic inference with continuous variables. In Proc. of the Conf. on Uncertainry in AI, pages 77-85, 1991.

Digital Library

[6]

K. C. Chang and R. M. Fung. Symbolic probabilistic inference with both discrete and continuous vatiables. IEEE Trans. on Systems, Man, and Cybernetics, 25(6):910-917, 1995.

[7]

R. Dechter. Bucket elimination: a unifying frame work for probabilistic inference. In M. Jordan, editor, Learning in Graphical Models. MIT Press, 1998.

Digital Library

[8]

E. Driver and D. Monel. Implementation of continuous Bayesian networks usings sums of weighted Gaussians. In Proc. of the Conf. on Uncertainty in AI, pages 134-140, 1995.

Digital Library

[9]

N. Friedman and M. Goldszmidt. Discretizing continuous attributes while learning bayesian networks. In Machine Learning: Proceedings of the International Conference, 1996.

[10]

W. Gilks, S. Richardson, and D. Spiegelhalter. Markov Chain Monte Carlso in Practice. Chapman and Hall, 1996.

[11]

C. Huang and A. Darwiche. Inference in belief networks: A procedural guide. Intl. J. Approx. Reasoning, 11, 1994.

[12]

F. Jensen and S. K. Andersen. Approximations in Bayesian belief universes for knowledge-based systems. In Proc. of the Conf. on Uncertainty in AI, 1990.

[13]

T. Jaakkola. Variational Methods for Inference and Estimation in Graphical Models. PhD thesis, MIT, 1997.

Digital Library

[14]

M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. In M. Jordan, editor, Learning in Graphical Models. MIT Press, 1998.

Digital Library

[15]

F. V. Jensen and F. Jensen. Optimal junction trees. In Proc. of the Conf. on Uncertainty in AI, 1994.

[16]

M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6:181-214, 1994.

Digital Library

[17]

T. Jaakkola and M. Jordan. A variational approach to Bayesian logistic regression problems and their extensions. In AI+ Statistics, 1996.

[18]

T.S. Jaakkola and M.I. Jordan. Variational probabilistic inference and the QMR-DT network. J. of AI Research, 10, 1999.

Digital Library

[19]

M. I. Jordan. Why the logistic function? A tutorial discussion on probabilities and neural networks. Technical Report 9503, MIT Computational Cognitive Science Report, August 1995.

[20]

U. Kjaemlff. Triangulation of graphs - algorithms giving small total state space. Technical Report R- 90-09, Dept. of Math. andComp. Sci., Aalborg Univ., Denmark, 1990. Available at www.cs.auc.dk/ uk.

[21]

A. V. Kozlov and D. Koller. Nonuniform dynamic discretization in hybrid networks. In Proc. of the Conf. on Uncertainty in AI, 1997.

Digital Library

[22]

S. L. Lauritzen. Propagation of probabilities, means and variances in mixed graphical association models. J. of the Am. Stat. Assoc., 87(420):1098-1108, December 1992.

[23]

S. Lauritzen. Graphical Models. OUP, 1996.

[24]

H.-G. Leimer. Triangulated graphs with marked vertices. Annals of Discrete Mathematics, 41:311-324, 1989.

[25]

S. L. Lauritzen and N. Wermuth. Graphical models for associations between variables, some of which are qualitative and some quantitative. Annals of statistics, 17:31-57, 1989.

[26]

R. J. McEliece and S. M. Aji. The generalized distributive law. IEEE Trans. Info. Theory, 1998. submitted.

Digital Library

[27]

McCullagh and Nelder. Generalized Linear Models. Chapman and Hall, 1983.

[28]

R. M. Neal and G. E. Hinton. A new view of the EM algorithm that justifies incremental and other variants. In M. Jordan, editor, Learning in Graphical Models. MIT Press, 1998.

Digital Library

[29]

K. G. Olesen. Causal probabilistic networks with both discrete andcontinuous variables. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3(15), 1993.

Digital Library

[30]

I. Rish and R. Dechter. On the impact of causal independence. Technical report, Information and Computer Science, UCI, 1998.

[31]

R. Shachter, S. Andersen, and P. Szolovits. Global conditioning for probabilistic inference in belief networks. In Proc. of the Conf: on Uncertainry in AI, 1994.

[32]

R. Shachterand C. R. Kenley. Gaussian influence diagrams. Managment Science, 35(5):527-550, 1989.

Digital Library

[33]

R. D. Shachter and M. A. Peot. Simulation approaches to general probabilistic inference on belief networks. In Proc. of the Conf. on Uncertainty in AI, volume 5, 1990.

Digital Library

[34]

M. Tipping. Probabilistic visualization of highdimensional binary data. In Neural Info. Proc. Systems, 1998.

Digital Library

Cited By

Salmerón ARumí RLangseth HNielsen TMadsen A(2018)A review of inference algorithms for hybrid Bayesian networksJournal of Artificial Intelligence Research10.1613/jair.1.1122862:1(799-828)Online publication date: 1-May-2018
https://dl.acm.org/doi/10.1613/jair.1.11228
Belle VVan Den Broeck GPasserini A(2016)Hashing-based approximate probabilistic inference in hybrid domainsProceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence10.5555/3061053.3061229(4115-4119)Online publication date: 9-Jul-2016
https://dl.acm.org/doi/10.5555/3061053.3061229
Belle VBroeck GPasserini A(2016)Component caching in hybrid domains with piecewise polynomial densitiesProceedings of the Thirtieth AAAI Conference on Artificial Intelligence10.5555/3016100.3016376(3369-3375)Online publication date: 12-Feb-2016
https://dl.acm.org/doi/10.5555/3016100.3016376
Show More Cited By

Index Terms

A variational approximation for Bayesian networks with discrete and continuous latent variables
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
      1. Probabilistic reasoning
      2. Vagueness and fuzzy logic

Recommendations

Mean-field variational approximate Bayesian inference for latent variable models

The ill-posed nature of missing variable models offers a challenging testing ground for new computational techniques. This is the case for the mean-field variational Bayesian inference. The behavior of this approach in the setting of the Bayesian probit ...
Latent-Space Variational Bayes

Variational Bayesian Expectation-Maximization (VBEM), an approximate inference method for probabilistic models based on factorizing over latent variables and model parameters, has been a standard technique for practical Bayesian inference. In this paper,...
Variational Bayesian Monte Carlo
NIPS'18: Proceedings of the 32nd International Conference on Neural Information Processing Systems

Many probabilistic models of interest in scientific computing and machine learning have expensive, black-box likelihoods that prevent the application of standard techniques for Bayesian inference, such as MCMC, which would require access to the gradient ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

UAI'99: Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence

July 1999

703 pages

ISBN:1558606149

Editors:
Kathryn B. Laskey
George Mason University, Fairfax, Virginia
,
Henri Prade
Institut de Recherche en Informatique de Toulouse, Toulouse, France

Sponsors

Rockwell Science Center: Rockwell Science Center
HUGIN: Hugin Expert A/S
Information Extraction and Transportation
Microsoft Research: Microsoft Research
AT&T: AT&T Labs Research

Publisher

Morgan Kaufmann Publishers Inc.

San Francisco, CA, United States

Publication History

Published: 30 July 1999

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
71
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)13

Reflects downloads up to 20 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Salmerón ARumí RLangseth HNielsen TMadsen A(2018)A review of inference algorithms for hybrid Bayesian networksJournal of Artificial Intelligence Research10.1613/jair.1.1122862:1(799-828)Online publication date: 1-May-2018
https://dl.acm.org/doi/10.1613/jair.1.11228
Belle VVan Den Broeck GPasserini A(2016)Hashing-based approximate probabilistic inference in hybrid domainsProceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence10.5555/3061053.3061229(4115-4119)Online publication date: 9-Jul-2016
https://dl.acm.org/doi/10.5555/3061053.3061229
Belle VBroeck GPasserini A(2016)Component caching in hybrid domains with piecewise polynomial densitiesProceedings of the Thirtieth AAAI Conference on Artificial Intelligence10.5555/3016100.3016376(3369-3375)Online publication date: 12-Feb-2016
https://dl.acm.org/doi/10.5555/3016100.3016376
Belle VVan den Broeck GPasserini A(2015)Hashing-based approximate probabilistic inference in hybrid domainsProceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence10.5555/3020847.3020863(141-150)Online publication date: 12-Jul-2015
https://dl.acm.org/doi/10.5555/3020847.3020863
Belle VPasserini AVan Den Broeck G(2015)Probabilistic inference in hybrid domains by weighted model integrationProceedings of the 24th International Conference on Artificial Intelligence10.5555/2832581.2832636(2770-2776)Online publication date: 25-Jul-2015
https://dl.acm.org/doi/10.5555/2832581.2832636
Shenoy PWest J(2011)Extended Shenoy--Shafer architecture for inference in hybrid bayesian networks with deterministic conditionalsInternational Journal of Approximate Reasoning10.1016/j.ijar.2011.02.00552:6(805-818)Online publication date: 1-Sep-2011
https://dl.acm.org/doi/10.1016/j.ijar.2011.02.005
Shenoy PWest J(2011)Inference in hybrid Bayesian networks using mixtures of polynomialsInternational Journal of Approximate Reasoning10.1016/j.ijar.2010.09.00352:5(641-657)Online publication date: 1-Jul-2011
https://dl.acm.org/doi/10.1016/j.ijar.2010.09.003
Grochowski DTang Y(2009)A machine learning approach for optimal disassembly planningInternational Journal of Computer Integrated Manufacturing10.1080/0951192080202417622:4(374-383)Online publication date: 1-Apr-2009
https://dl.acm.org/doi/10.1080/09511920802024176
Langseth HNielsen T(2009)Latent classification models for binary dataPattern Recognition10.1016/j.patcog.2009.05.00242:11(2724-2736)Online publication date: 1-Nov-2009
https://dl.acm.org/doi/10.1016/j.patcog.2009.05.002
Raiko TValpola HHarva MKarhunen J(2007)Building Blocks for Variational Bayesian Learning of Latent Variable ModelsThe Journal of Machine Learning Research10.5555/1248659.12486658(155-201)Online publication date: 1-May-2007
https://dl.acm.org/doi/10.5555/1248659.1248665
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten