Article

Approximate factorizations of distributions and the minimum relative entropy principle

Authors:

Heinz Mühlenbein,

Robin HönsAuthors Info & Claims

GECCO '05: Proceedings of the 7th annual workshop on Genetic and evolutionary computation

Pages 199 - 211

https://doi.org/10.1145/1102256.1102306

Published: 25 June 2005 Publication History

Abstract

Estimation of Distribution Algorithms (EDA) have been proposed as an extension of genetic algorithms. In this paper the major design issues of EDA's are discussed within a general interdisciplinary framework, the maximum entropy approximation. Our EDA algorithm FDA assumes that the function to be optimized is additively decomposed (ADF). The interaction graph G_ADF is used to create exact or approximate factorizations of the Boltzmann distribution. The relation between FDA factorizations and the MaxEnt solution is shown. We introduce a second algorithm, derived from the Bethe-Kikuchi approach developed in statistical physics. It tries to minimize the Kullback-Leibler divergence KLD(q\pβ) to the Boltzmann distribution pβ by solving a difficult constrained optimization problem. We present in detail the concave-convex minimization algorithm CCCP to solve the optimization problem. The two algorithms are compared using popular benchmark problems (2-d grid problems, 2-d Ising spin glasses, Kaufman's n — k function.) We use instances up to 900 variables.

References

[1]

S. M. Aji and R. J. McEliece. The generalized distributive law and free energy minimization. In Proceedings of the 39th Annual Allerton Conference on Communication, Control, and Computing, pages 672--681, 2001.

[2]

R. G. Almond. Graphical Belief Modelling. Chapman & Hall, London, 1995.

Digital Library

[3]

U. Bertelè and F. Brioschi. Nonserial Dynamic Programming. Academic Press, New York, 1972.

Digital Library

[4]

T. M. Cover and J. Thomas. Elements of Information Theory. Wiley, New York, 1989.

Digital Library

[5]

Y. Gao and J. Culberson. Space complexity of estimation of distribution algorithms. Evolutionary Computation, 13(1):125--143, 2005.

Digital Library

[6]

R. Höns. Estimation of Distribution Algorithms and Minimum Relative Entropy. PhD thesis, University of Bonn, 2005.

[7]

E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev, 6:620--643, 1957.

[8]

E. T. Jaynes. Where do we stand on maximum entropy? In R. D. Levine and M. Tribus, editors, The Maximum Entropy Formalism. MIT Press, Cambridge, 1978.

[9]

F. V. Jensen and F. Jensen. Optimal junction trees. In Proceedings of the 10th Conference on Uncertainty in Artificial Intelligence, pages 360--366, Seattle, 1994.

[10]

M. I. Jordan, editor. Learning in Graphical Models. MIT Press, Cambrigde, 1999.

Digital Library

[11]

S. L. Lauritzen. Graphical Models. Clarendon Press, Oxford, 1996.

[12]

A. Martelli and U. Montanari. Nonserial dynamic programming: On the optimal strategy of variable elimination for the rectangular lattice. J. Math. Anal. Appl., 40:226--242, 1972.

[13]

R. J. McEliece and M. Yildirim. Belief propagation on partially ordered sets. In Proceedings of the 15th International Symposium on Mathematical Theory of Networks and Systems (MTNS 2002), 2002.

[14]

H. Mühlenbein and R. Höns. The estimation of distributions and the minimum relative entropy principle. Evolutionary Computation, 13(1):1--27, 2005.

Digital Library

[15]

H. Mühlenbein and T. Mahnig. FDA - a scalable evolutionary algorithm for the optimization of additively decomposed functions. Evolutionary Computation, 7(4):353--376, 1999.

Digital Library

[16]

H. Mühlenbein and T. Mahnig. Evolutionary optimization and the estimation of search distributions with applications to graph bipartitioning. Journal of Approximate Reasoning, 31(3):157--192, 2002.

[17]

H. Mühlenbein and T. Mahnig. Mathematical analysis of evolutionary algorithms. In C. C. Ribeiro and P. Hansen, editors, Essays and Surveys in Metaheuristics, Operations Research/Computer Science Interface Series, pages 525--556. Kluwer Academic Publisher, Norwell, 2002.

[18]

H. Mühlenbein and T. Mahnig. Evolutionary algorithms and the Boltzmann distribution. In K. D. Jong, R. Poli, and J. C. Rowe, editors, Foundations of Genetic Algorithms 7, pages 525--556. Morgan Kaufmann Publishers, San Francisco, 2003.

[19]

H. Mühlenbein, T. Mahnig, and A. Ochoa. Schemata, distributions and graphical models in evolutionary optimization. Journal of Heuristics, 5(2):213--247, 1999.

Digital Library

[20]

H. Mühlenbein and G. Paaß. From recombination of genes to the estimation of distributions I. binary parameters. In H.-M. Voigt, W. Ebeling, I. Rechenberg, and H.-P. Schwefel, editors, Lecture Notes in Computer Science 1141: Parallel Problem Solving from Nature - PPSN IV), pages 178--187, Berlin, 1996. Springer-Verlag.

Digital Library

[21]

M. Pelikan and D. Goldberg. Hierarchical BOA solves Ising spin glasses and MAXSAT. In Genetic and Evolutionary Computation Conference 2003, volume 2724 of Lecture Notes in Computer Science, pages 1271--1282. Springer, 2003. Also IlliGAL Report No. 2003001.

Digital Library

[22]

R. Santana. Estimation of distribution algorithms with Kikuchi approximations. Evolutionary Computation, 13(1):67--97, 2005.

Digital Library

[23]

Y. W. Teh and M. Welling. On improving the efficiency of the iterative proportional fitting procedure. In Proceedings of the International Workshop on Artificial Intelligence and Statistics, volume 9, 2003.

[24]

Y. Xiang, S. K. M. Wong, and N. Cercone. A 'microscopic' study of minimum entropy search in learning decomposable Markov networks. Machine Learning, 26:65--92, 1997.

Digital Library

[25]

C. Yanover and Y. Weiss. Finding the M most probable configurations using loopy belief propagation. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16. MIT Press, Cambridge, MA, 2004.

[26]

J. S. Yedidia, W. T. Freeman, and Y. Weiss. Constructing free energy approximations and generalized belief propagation algorithms. Technical Report 2004--040, Mitsubishi Electric Research Laboratories, May 2004.

[27]

A. L. Yuille. CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation. Neural Computation, 14(7):1691--1722, 2002.

Digital Library

Cited By

Gwizdałła T(2012)The role of different genetic operators in the optimization of magnetic modelsApplied Mathematics and Computation10.1016/j.amc.2012.02.078218:18(9220-9233)Online publication date: May-2012
https://doi.org/10.1016/j.amc.2012.02.078
Gwizdałła TLanzi P(2011)Different versions of particle swarm optimization for magnetic problemsProceedings of the 13th annual conference companion on Genetic and evolutionary computation10.1145/2001858.2001862(5-6)Online publication date: 12-Jul-2011
https://dl.acm.org/doi/10.1145/2001858.2001862

Index Terms

Approximate factorizations of distributions and the minimum relative entropy principle
1. Theory of computation
  1. Design and analysis of algorithms

Recommendations

Maximum entropy characterizations of the multivariate Liouville distributions

A random vector X = (X₁, X₂,...,X_n) with positive components has a Liouville distribution with parameter θ = (θ₁, θ₂,...,θ_n) if its joint probability density function is proportional to h(Σ_i=1ⁿ x_i) Φ_i=1ⁿ x_i^θ_i-1, θ_i > 0 [R.D. Gupta, D.S.P. Richards, ...
On some entropy functionals derived from Rényi information divergence

We consider the maximum entropy problems associated with Renyi Q-entropy, subject to two kinds of constraints on expected values. The constraints considered are a constraint on the standard expectation, and a constraint on the generalized expectation as ...
Fda -a scalable evolutionary algorithm for the optimization of additively decomposed functions

The Factorized Distribution Algorithm (FDA) is an evolutionary algorithm which combines mutation and recombination by using a distribution. The distribution is estimated from a set of selected points. In general, a discrete distribution defined for n ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '05: Proceedings of the 7th annual workshop on Genetic and evolutionary computation

June 2005

431 pages

ISBN:9781450378000

DOI:10.1145/1102256

Conference Chair:
Franz Rothlauf
University of Mannheim, Germany

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

GECCO05

Sponsor:

SIGEVO

GECCO05: Genetic and Evolutionary Computation Conference

June 25 - 26, 2005

D.C., Washington

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
227
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gwizdałła T(2012)The role of different genetic operators in the optimization of magnetic modelsApplied Mathematics and Computation10.1016/j.amc.2012.02.078218:18(9220-9233)Online publication date: May-2012
https://doi.org/10.1016/j.amc.2012.02.078
Gwizdałła TLanzi P(2011)Different versions of particle swarm optimization for magnetic problemsProceedings of the 13th annual conference companion on Genetic and evolutionary computation10.1145/2001858.2001862(5-6)Online publication date: 12-Jul-2011
https://dl.acm.org/doi/10.1145/2001858.2001862

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten