research-article

Genetic programming benchmarks: looking back and looking forward

Authors:

James McDermott,

Gabriel Kronberger,

Patryk Orzechowski,

Leonardo Vanneschi,

Roman Kalkreuth,

Mauro CastelliAuthors Info & Claims

ACM SIGEVOlution, Volume 15, Issue 3

Article No.: 1, Pages 1 - 19

https://doi.org/10.1145/3578482.3578483

Published: 23 December 2022 Publication History

Abstract

The top image shows a set of scales, which are intended to bring to mind the ideas of balance and fair experimentation which are the focus of our article on genetic programming benchmarks in this issue. Image by Elena Mozhvilo and made available under the Unsplash license on https://unsplash.com/photos/j06gLuKK0GM.

References

[1]

Akiba, T., Sano, S., Yanase, T., Ohta, T. and Koyama, M., 2019, July. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2623--2631).

[2]

Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2017). Deepcoder: Learning to write programs. In International Conference on Learning Representations. ICLR

[3]

Burke, E. K., et al. "Towards the decathlon challenge of search heuristics." Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers. 2009.

[4]

Burnham, K. P. Model selection and multimodel inference. A practical information-theoretic approach (1998).

[5]

Chen, X., Liu, C., & Song, D. (2018,). Execution-guided neural program synthesis. In International Conference on Learning Representations. ICLR

[6]

Christie, L.A., Brownlee, A.E. and Woodward, J.R., 2018, July. Investigating benchmark correlations when comparing algorithms with parameter tuning. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 209--210).

[7]

Cranmer, M., PySR: Fast & parallelized symbolic regression in Python/Julia, September 2020.

[8]

Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A. R., & Kohli, P. (2017). Robustfill: Neural program learning under noisy i/o. In International conference on machine learning (pp. 990--998). ICML.

[9]

Forsyth, R., 1981. BEAGLE - A Darwinian approach to pattern recognition. Kybernetes.

[10]

GitHub Team, 2022. GitHub Copilot. https://github.com/features/copilot

[11]

Gu, K., Liu, M., Zhai, G., Yang, X. and Zhang, W., 2015. Quality assessment considering viewing distance and image resolution. IEEE Transactions on Broadcasting, 61(3), pp.520--531.

[12]

Helmuth, T. and Kelly, P., 2021, June. PSB2: the second program synthesis benchmark suite. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 785--794).

[13]

Helmuth, T. and Spector, L., 2015, July. General program synthesis benchmark suite. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (pp. 1039--1046).

[14]

Izzo, D., Biscani, F. and Mereta, A., 2017, April. Differentiable genetic programming. In European conference on genetic programming (pp. 35--51). Springer, Cham.

[15]

Koza, J. R., 1992. Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press.

Digital Library

[16]

Lehman, J., et al. "The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities." Artificial life 26.2 (2020): 274--306.

[17]

Liao, T., Taori, R., Raji, I.D. and Schmidt, L., 2021, August. Are we learning yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

[18]

McDermott, J. "When and why metaheuristics researchers can ignore "No Free Lunch" theorems." SN Computer Science 1.1 (2020): 1--18.

[19]

McDermott, J. and Forsyth, R.S., 2016. Diagnosing a disorder in a classification benchmark. Pattern Recognition Letters, 73, pp.41--43.

Digital Library

[20]

Miller, J.F., 2020. Cartesian genetic programming: its status and future. Genetic Programming and Evolvable Machines, 21(1), pp.129--168.

Digital Library

[21]

Moravec, H (1988), Mind Children, Harvard University Press

[22]

Nicolau, M., Agapitos, A., O'Neill, M., & Brabazon, A. (2015, May). Guidelines for defining benchmark problems in genetic programming. In 2015 IEEE Congress on Evolutionary Computation (CEC) (pp. 1152--1159). IEEE.

[23]

Nicolau, M., 2017. Understanding grammatical evolution: initialisation. Genetic Programming and Evolvable Machines, 18(4), pp.467--507.

Digital Library

[24]

Oliveira, L.O.V., Martins, J.F.B., Miranda, L.F. and Pappa, G.L., 2018, July. Analysing symbolic regression benchmarks under a meta-learning approach. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 1342--1349).

[25]

Olson, R. S., William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. 2017. PMLB: a large benchmark suite for machine learning evaluation and comparison. BioData Mining 10, 36 (11 Dec 2017), 1--13.

[26]

Parisotto, E., Mohamed, A. R., Singh, R., Li, L., Zhou, D., & Kohli, P. (2017). In International Conference on Learning Representations. ICLR.

[27]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V. and Vanderplas, J., 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, pp.2825--2830.

[28]

Ponomarenko, N., Jin, L., Ieremeiev, O., Lukin, V., Egiazarian, K., Astola, J., Vozel, B., Chehdi, K., Carli, M., Battisti, F. and Kuo, C.C.J., 2015. Image database TID2013: Peculiarities, results and perspectives. Signal processing: Image communication, 30, pp.57--77.

[29]

Roelofs, R., Shankar, V., Recht, B., Fridovich-Keil, S., Hardt, M., Miller, J. and Schmidt, L., 2019. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems, 32.

[30]

Ryan, C., Collins, J.J. and O'Neill, M., 1998, April. Grammatical evolution: Evolving programs for an arbitrary language. In European conference on genetic programming (pp. 83--96). Springer, Berlin, Heidelberg.

[31]

Salustowicz, R. and Schmidhuber, J., 1997. Probabilistic incremental program evolution. Evolutionary computation, 5(2), pp.123--141.

[32]

Schmidt, M. and Lipson, H., 2009. Distilling free-form natural laws from experimental data. science, 324(5923), pp.81--85.

[33]

Sobania, D., M. Briesch, and F. Rothlauf. "Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming." Proceedings of the Genetic and Evolutionary Computation Conference. 2022.

Digital Library

[34]

Stephens, T., 2016. Genetic Programming in Python With a Scikit-Learn Inspired API: gplearn. https://gplearn.readthedocs.io.

[35]

Udrescu, S.-M., and M. Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances 6.16 (2020): eaay2631.

[36]

Urbanowicz, R. J., J. Kiralis, N. A. Sinnott-Armstrong, T. Heberling, J. M. Fisher, and J. H. Moore. "GAMETES: a fast, direct algorithm for generating pure, strict, epistatic models with random architectures." BioData mining 5, no. 1 (2012): 1--14.

[37]

Vanschoren, J., Van Rijn, J.N., Bischl, B. and Torgo, L., 2014. OpenML: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2), pp.49--60.

Digital Library

[38]

Vladislavleva, E.J., Smits, G.F. and Den Hertog, D., 2008. Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming. IEEE Transactions on Evolutionary Computation, 13(2), pp.333--349.

Digital Library

Cited By

Cárdenas Florido LTrujillo LHernandez DMuñoz Contreras J(2024)M5GP: Parallel Multidimensional Genetic Programming with Multidimensional Populations for Symbolic RegressionMathematical and Computational Applications10.3390/mca2902002529:2(25)Online publication date: 18-Mar-2024
https://doi.org/10.3390/mca29020025
Nadizar GSakallioglu BGarrow FSilva SVanneschi L(2024)Geometric semantic GP with linear scaling: Darwinian versus Lamarckian evolutionGenetic Programming and Evolvable Machines10.1007/s10710-024-09488-025:2Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10710-024-09488-0
Kalkreuth RVašíček ZHusa JVermetten DYe FBäck T(2023)General Boolean Function Benchmark SuiteProceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms10.1145/3594805.3607131(84-95)Online publication date: 30-Aug-2023
https://dl.acm.org/doi/10.1145/3594805.3607131
Show More Cited By

Recommendations

Genetic programming needs better benchmarks
GECCO '12: Proceedings of the 14th annual conference on Genetic and evolutionary computation

Genetic programming (GP) is not a field noted for the rigor of its benchmarking. Some of its benchmark problems are popular purely through historical contingency, and they can be criticized as too easy or as providing misleading information concerning ...
Benchmarks that matter for genetic programming
GECCO Comp '14: Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation

There have been several papers published relating to the practice of benchmarking in machine learning and Genetic Programming (GP) in particular. In addition, GP has been accused of targeting over-simplified 'toy' problems that do not reflect the ...
An Introduction to Information Security: NIST 800-12 Rev 1

Comments

Information & Contributors

Information

Published In

cover image ACM SIGEVOlution

ACM SIGEVOlution Volume 15, Issue 3

September 2022

19 pages

EISSN:1931-8499

DOI:10.1145/3578482

Editor:
Gabriela Ochoa

Issue’s Table of Contents

Copyright © 2022 Copyright is held by the owner/author(s).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 December 2022

Published in SIGEVO Volume 15, Issue 3

Check for updates

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
104
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)4

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cárdenas Florido LTrujillo LHernandez DMuñoz Contreras J(2024)M5GP: Parallel Multidimensional Genetic Programming with Multidimensional Populations for Symbolic RegressionMathematical and Computational Applications10.3390/mca2902002529:2(25)Online publication date: 18-Mar-2024
https://doi.org/10.3390/mca29020025
Nadizar GSakallioglu BGarrow FSilva SVanneschi L(2024)Geometric semantic GP with linear scaling: Darwinian versus Lamarckian evolutionGenetic Programming and Evolvable Machines10.1007/s10710-024-09488-025:2Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10710-024-09488-0
Kalkreuth RVašíček ZHusa JVermetten DYe FBäck T(2023)General Boolean Function Benchmark SuiteProceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms10.1145/3594805.3607131(84-95)Online publication date: 30-Aug-2023
https://dl.acm.org/doi/10.1145/3594805.3607131
Farinati DBakurov IVanneschi L(2023)A study of dynamic populations in geometric semantic genetic programmingInformation Sciences: an International Journal10.1016/j.ins.2023.119513648:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.ins.2023.119513

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents