Abstract
A Bayesian framework for genetic programming (GP) is presented. This is motivated by the observation that genetic programming iteratively searches populations of fitter programs and thus the information gained in the previous generation can be used in the next generation. The Bayesian GP makes use of Bayes theorem to estimate the posterior distribution of programs from their prior distribution and likelihood for the fitness data observed. Offspring programs are then generated by sampling from the posterior distribution by genetic variation operators. We present two GP algorithms derived from the Bayesian GP framework. One is the genetic programming with the adaptive Occam's razor (AOR) designed to evolve parsimonious programs. The other is the genetic programming with incremental data inheritance (IDI) designed to accelerate evolution by active selection of fitness cases. A multiagent learning task is used to demonstrate the effectiveness of the presented methods. In a series of experiments, AOR reduced solution complexity by 20% and IDI doubled evolution speed, both without loss of solution accuracy.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
D. Andre, “Automatically defined features: The simultaneous evolution of 2-dimensional feature detectors and an algorithm for using them,” In K. E. Kinnear, Jr. ed, Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994, Chapter 23, 0pp. 477
P. J. Angeline and J. B. Pollack, “Coevolving high-level representations, in Artificial Life III, C. G. Langton ed., Addison-Wesley: Reading, MA, 1993.
P. J. Angeline, “Genetic programming and emergent intelligence,” in K. E. Kinnear, Jr. ed., Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994. Chapter 4, pp. 75–98.
P. J. Angeline, “Subtree crossover: Building block engine or macromutation?” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann, San Francisco, CA, 1997, pp. 9–17.
T. Back, Evolutionary Algorithms in Theory and Practice, Oxford, UK: Oxford University Press, 1996.
W. Banzhaf, F. Francone, and P. Nordin, “The effect of extensive use of the mutation operator on generalization in genetic programming using sparse data sets,” In Proc. 4th Int. Conf. on Parallel Problem Solving from Nature PPSN-96, W. Ebeling, I. Rechenberg, H.-P. Schwefel, H. M. Voigt eds., Springer: Berlin, 1996, pp. 300–309.
W. Banzhaf, P. Nordin, and F. Francone, “On some emergent properties of variable size evolutionary algorithms,” in ICGA-97 Workshop on Evolutionary Computation with Variable-Size Representation, 1997, http:rrwww.ai.mit.edurpeoplerunamayricga-ws.html.
W. Banzhaf, P. Nordin, R. Keller, and F. Francone, Genetic Programming: An Introduction, Morgan Kaufmann: San Francisco, CA.
C. M. Bishop, Neural Networks for Pattern Recognition. Oxford University Press: Oxford, UK, 1995.
T. Blickle, “Evolving compact solutions in genetic programming: A case study,” in H.-M. Voigt et al. eds., Parallel Problem Solving from Nature IV, Springer-Verlag: Berlin, 1996, pp. 564–573.
K. Chellapilla, “Evolutionary programming with tree mutations: Evolving computer programs without crossover,” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann: San Francisco, CA, 1997, pp. 431–438.
T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley: New York, 1991.
C. Gathercole and P. Ross, “Dynamic training subset selection for supervised learning in genetic programming,” in Parallel Problem Solving from Nature III, Y. Davidor, et al. Eds., Springer-Verlag: Berlin, 1994, pp. 312–321.
C. Gathercole and P. Ross, “Small populations over many generations can beat large populations over few generations in genetic programming,” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann: San Francisco, CA, 1997, pp. 111–118.
A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin, Bayesian Data Analysis, Chapman & Hall: London, 1995.
T. Haynes, “Perturbing the representation, decoding, and evaluation of chromosomes,” in J. R. Koza, et al. eds., The Third Genetic Programming Conf. GP-98, Morgan Kaufmann: San Francisco, CA, 1998, pp. 122–127.
D. Hillis, “Co-evolving parasites improves simulated evolution as an optimization procedure,” in Artificial Life II, C. Langton, et al. Eds., Addison-Wesley: Reading, MA, 1992, pp. 313–324.
H. Iba, H. de Garis, and T. Sato, “Genetic programming using a minimum description length principle,” in K. E. Kinnear, Jr. ed., Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994, Chapter 12, pp. 265–284.
K. E. Kinnear, Jr. “Generality and difficulty in genetic programming: Evolving a sort,” in Proc. of 5th Int. Conf. on Genetic Algorithms ICGA-93, S. Forrest ed., Morgan Kaufmann: San Francisco, CA, 1993, pp. 287–294.
J. R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, MIT Press: Cambridge, MA, 1992.
J. R. Koza, “Scalable learning in genetic programming using automatic function definition,” in K. E. Kinnear, Jr. ed., Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994, Chapter 5, pp. 99–117.
W. B. Langdon and R. Poli, “Fitness causes bloat: Mutation,” in W. Banzhaf, R. Poli, M. Schoenauer, and T. Fogarty eds., The First European Workshop on Genetic Programming EuroGP'98, Paris, LNCS 1391, Springer-Verlag: Berlin, 1998, pp. 37–48.
W. B. Langdon, “Size fair and homologous tree crossovers,” Genetic Programming and Evolvable Machines, vol. 1(1), pp. 95–119, 2000.
S. Luke and L. Spector, “A comparison of crossover and mutation in genetic programming,” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann: San Francisco, CA, 1997, pp. 240–248.
H. Muhlenbein and D. Schlierkamp-Voosen, “The science of breeding and its application to the breeder genetic algorithm,” Evolutionary Computation, vol. 1(4) pp. 335–360, 1994.
R. M. Neal, “Probabilistic inference using Markov chain Monte Carlo methods,” Technical Report CRG-TR–93-1, Dept. of Computer Science, University of Toronto, 1993.
P. Nordin, F. Francone, and W. Banzhaf, “Explicitly defined introns and destructive crossover in genetic programming,” in P. J. Angeline and K. E. Kinnear, Jr. eds., Advances in Genetic Programming 2, MIT Press: Cambridge, MA, 1996, pp. 111–134.
U.-M. O'Reilly, “Investigating the generality of automatically defined functions,” in The First Genetic Programming Conf. GP-96, J. R. Koza eds., Morgan Kaufmann: San Francisco, CA, 1996, pp. 351–356.
R. Poli, and W. B. Langdon, “On the search properties of different crossover operators in genetic programming,” The Third Genetic Programming Conf. GP-98, Morgan Kaufmann: San Francisco, CA, 1998, pp. 293–301.
J. Rissanen, “Stochastic complexity and modeling,” Ann. Statist. vol. 14, pp. 1080–1100, 1986.
J. P. Rosca, “Analysis of complexity drift in genetic programming,” in The Second Genetic Programming Conf. GP-97, J. R. Koza eds., Morgan Kaufmann: San Francisco, CA, 1997, pp. 286–294.
J. P. Rosca and D. H. Ballard, “Discovery of subroutines in genetic programming,” in P. J. Angeline and K. E. Kinnear, Jr. ed., Advances in Genetic Programming 2, MIT Press: Cambridge, MA, 1996, pp. 177–201.
M. Schoenauer, M. Sebag, F. Jouve, B. Lamy, and H. Maitournam, “Evolutionary identification of macro-mechanical models,” in Advances in Genetic Programming 2, P. J. Angeline and K. E. Kinnear, Jr. eds., MIT Press: Cambridge, MA, 1996, pp. 467–488.
E. V. Siegel, “Competitively evolving decision trees against fixed training cases for natural language processing,” in Advances in Genetic Programming, K. E. Kinnear, Jr. ed., MIT Press: Cambridge, MA, 1994, Chapter 19, pp. 409–423.
T. Soule, J. A. Foster, and J. Dickinson, “Code growth in genetic programming,” in The First Genetic Programming Conf. GP-96, J. P. Koza, et al. eds., Morgan Kaufmann: San Francisco, CA, 1996, pp. 215–223.
T. Soule and J. A. Foster, “Effects of code growth and parsimony pressure on populations in genetic programming,” Evolutionary Computation vol. 6(4), pp. 293–309, 1998.
L. Spector, “Simultaneous evolution of programs and their control structures,” in Advances in Genetic Programming 2, P. J. Angeline and K. E. Kinnear, Jr. eds., MIT Press: Cambridge, MA, 1996, Chapter 7, pp. 137–154.
A. Teller and D. Andre, “Automatically choosing the number of fitness cases: The rational allocation of trials,” in The Second Genetic Programming Conf. GP-97, J. R. Koza, et al. eds., Morgan Kaufmann: San Francisco, CA, 1997, pp. 321–328.
A. S. Wu and R. K. Lindsay, “Empirical studies of the genetic algorithm with noncoding segments,” Evolutionary Computation, vol. 3(2), pp. 121–147, 1996.
B.-T. Zhang, “Accelerated learning by active example selection,” Int. J. Neural Syst. vol. 5(4) pp. 67–75, 1994.
B.-T. Zhang, “A Bayesian framework for evolutionary computation,” in The 1999 Congress on Evolutionary Computation CEC99, Special Session on Theory and Foundations of Evolutionary Computation, IEEE Press, 1999, pp. 722–727.
B.-T. Zhang and J.-G. Joung, “Genetic programming with incremental data inheritance,” In The 1999 Genetic and Evolutionary Computation Conf. GECCO-99, W. Banzhaf et al. eds., Morgan Kaufmann: San Francisco, CA, 1999, pp. 1217–1224.
B.-T. Zhang and H. Muhlenbein, “Genetic programming of minimal neural nets using Occam's razor,” in Proc. of 5th Int. Conf. on Genetic Algorithms ICGA-93, S. Forrest ed., Morgan Kaufmann: San Francisco, CA, 1993, pp. 342–349.
B.-T. Zhang and H. Muhlenbein, “Balancing accuracy and parsimony in genetic programming,” Evolutionary Computation, vol. 3(1) pp. 17–38, 1995.
B.-T. Zhang, P. Ohm, and H. Muhlenbein, “Evolutionary induction of sparse neural trees,” Evolutionary Computation, Vol. 5(1) pp. 213–236, 1997.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Zhang, BT. Bayesian Methods for Efficient Genetic Programming. Genetic Programming and Evolvable Machines 1, 217–242 (2000). https://doi.org/10.1023/A:1010010230007
Issue Date:
DOI: https://doi.org/10.1023/A:1010010230007