Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3594805.3607131acmconferencesArticle/Chapter ViewAbstractPublication PagesfogaConference Proceedingsconference-collections
research-article
Open access

General Boolean Function Benchmark Suite

Published: 30 August 2023 Publication History

Abstract

Just over a decade ago, the first comprehensive review on the state of benchmarking in Genetic Programming (GP) analyzed the mismatch between the problems that are used to test the performance of GP systems and real-world problems. Since then, several benchmark suites in major GP problem domains have been proposed over time, filling some of the major gaps. In the framework of the first review about the state of benchmarking in GP, logic synthesis (LS) was classified as one of the major GP problem domains. However, a diverse and accessible benchmark suite for LS is still missing. In this work, we propose a benchmark suite for LS that covers different types of Boolean functions that are commonly used in the field of GP. We analyze the complexity of the proposed benchmark by using popular complexity measures that are commonly used to classify and characterize Boolean functions and digital circuits.

References

[1]
Timothy Atkinson, Detlef Plump, and Susan Stepney. 2018. Evolving Graphs by Graph Programming. In Genetic Programming - 21st European Conference, EuroGP 2018, Parma, Italy, April 4-6, 2018, Proceedings (Lecture Notes in Computer Science, Vol. 10781), Mauro Castelli, Lukás Sekanina, Mengjie Zhang, Stefano Cagnoni, and Pablo García-Sánchez (Eds.). Springer, 35--51. https://doi.org/10.1007/978-3-319-77553-1_3
[2]
Thomas Bartz-Beielstein, Carola Doerr, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, Manuel López-Ibáñez, Katherine M. Malan, Jason H. Moore, Boris Naujoks, Patryk Orzechowski, Vanessa Volz, Markus Wagner, and Thomas Weise. 2020. Benchmarking in Optimization: Best Practice and Open Issues. CoRR abs/2007.03488 (2020). arXiv:2007.03488 https://arxiv.org/abs/2007.03488
[3]
Lashon B. Booker, David E. Goldberg, and John H. Holland. 1989. Classifier Systems and Genetic Algorithms. Artif. Intell. 40, 1-3 (1989), 235--282. https://doi.org/10.1016/0004-3702(89)90050-7
[4]
Ann Braeken. 2006. Cryptographic properties of Boolean functions and S-boxes. Ph.D. Dissertation. Katholieke Universiteit Leuven.
[5]
A. Canteaut and M. Videau. 2005. Symmetric Boolean functions. IEEE Transactions on Information Theory 51, 8 (2005), 2791--2811. https://doi.org/10.1109/TIT.2005.851743
[6]
Claude Carlet. 2010. Boolean Functions for Cryptography and Error-Correcting Codes. Cambridge University Press, 257--397. https://doi.org/10.1017/CBO9780511780448.011
[7]
Claude Carlet and Xi Chen. 2017. Constructing low-weight d th-order correlation-immune Boolean functions through the Fourier-Hadamard transform. IEEE Transactions on Information Theory 64, 4 (2017), 2969--2978.
[8]
Nichael Lynn Cramer. 1985. A Representation for the Adaptive Generation of Simple Sequential Programs. In Proceedings of the 1st International Conference on Genetic Algorithms, Pittsburgh, PA, USA, July 1985, John J. Grefenstette (Ed.). Lawrence Erlbaum Associates, 183--187.
[9]
Jacob de Nobel, Furong Ye, Diederick Vermetten, Hao Wang, Carola Doerr, and Thomas Bäck. 2021. IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics. CoRR abs/2111.04077 (2021). arXiv:2111.04077 https://arxiv.org/abs/2111.04077
[10]
Lucas Augusto Müller de Souza, José Eduardo Henriques da Silva, Luciano Jerez Chaves, and Heder Soares Bernardino. 2020. A benchmark suite for designing combinational logic circuits via metaheuristics. Appl. Soft Comput. 91 (2020), 106246. https://doi.org/10.1016/j.asoc.2020.106246
[11]
Richard Forsyth. 1981. BEAGLE A Darwinian Approach to Pattern Recognition. Kybernetes 10, 3 (1981), 159--166. https://doi.org/
[12]
Gary D. Hachtel and Fabio Somenzi. 1996. Logic synthesis and verification algorithms. Kluwer.
[13]
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tusar, and Dimo Brockhoff. 2021. COCO: a platform for comparing continuous optimizers in a black-box setting. Optim. Methods Softw. 36, 1 (2021), 114--144. https://doi.org/10.1080/10556788.2020.1808977
[14]
S. Harding, J. F. Miller, and W. Banzhaf. 2009. Self modifying Cartesian Genetic Programming: Parity. In 2009 IEEE Congress on Evolutionary Computation. 285--292. https://doi.org/10.1109/CEC.2009.4982960
[15]
Soha Hassoun and Tsutomu Sasao. 2001. Logic synthesis and verification. Vol. 654. Springer Science & Business Media.
[16]
Thomas Helmuth and Peter Kelly. 2021. PSB2: the second program synthesis benchmark suite. In GECCO '21: Genetic and Evolutionary Computation Conference, Lille, France, July 10-14, 2021, Francisco Chicano and Krzysztof Krawiec (Eds.). ACM, 785--794. https://doi.org/10.1145/3449639.3459285
[17]
Thomas Helmuth and Lee Spector. 2015. General Program Synthesis Benchmark Suite. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2015, Madrid, Spain, July 11-15, 2015, Sara Silva and Anna Isabel Esparcia-Alcázar (Eds.). ACM, 1039--1046. https://doi.org/10.1145/2739480.2754769
[18]
Joseph Hicklin. 1986. Application of the Genetic Algorithm to Automatic Program Generation. Master's thesis. University of Idaho.
[19]
David Hodan, Vojtech Mrazek, and Zdenek Vasicek. 2020. Semantically-Oriented Mutation Operator in Cartesian Genetic Programming for Evolutionary Circuit Design. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference (Cancún, Mexico) (GECCO '20). Association for Computing Machinery, New York, NY, USA, 940--948. https://doi.org/10.1145/3377930.3390188
[20]
David Hodan, Vojtech Mrazek, and Zdenek Vasícek. 2021. Semantically-oriented mutation operator in cartesian genetic programming for evolutionary circuit design. Genet. Program. Evolvable Mach. 22, 4 (2021), 539--572. https://doi.org/10.1007/s10710-021-09416-6
[21]
John H. Holland. 1987. Genetic Algorithms and Classifier Systems: Foundations and Future Directions. In Proceedings of the 2nd International Conference on Genetic Algorithms, Cambridge, MA, USA, July 1987, John J. Grefenstette (Ed.). Lawrence Erlbaum Associates, 82--89.
[22]
John H. Holland and Judith S. Reitman. 1977. Cognitive systems based on adaptive algorithms. SIGART Newsl. 63 (1977), 49. https://doi.org/10.1145/1045343.1045373
[23]
Radek Hrbacek and Lukas Sekanina. 2014. Towards Highly Optimized Cartesian Genetic Programming: From Sequential via SIMD and Thread to Massive Parallel Implementation. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation (Vancouver, BC, Canada) (GECCO 14). Association for Computing Machinery, New York, NY, USA, 1015--1022. https://doi.org/10.1145/2576768.2598343
[24]
Jakub Husa. 2019. Comparison of genetic programming methods on design of cryptographic boolean functions. In European Conference on Genetic Programming. Springer, 228--244.
[25]
Jakub Husa and Lukas Sekanina. 2020. Evolving Cryptographic Boolean Functions with Minimal Multiplicative Complexity. In 2020 IEEE Congress on Evolutionary Computation (CEC). IEEE, 1--8.
[26]
Domagoj Jakobovic, Stjepan Picek, Marcella SR Martins, and Markus Wagner. 2021. Toward more efficient heuristic construction of Boolean functions. Applied Soft Computing 107 (2021), 107327.
[27]
T. Kalganova. 1997. Evolutionary Approach to Design Multiple-valued Combinational Circuits. In Proceedings. of the 4th International conference on Applications of Computer Systems (ACS'97). Szczecin, Poland, 333--339.
[28]
Roman Kalkreuth. 2019. Two New Mutation Techniques for Cartesian Genetic Programming. In Proceedings of the 11th International Joint Conference on Computational Intelligence, IJCCI 2019, Vienna, Austria, September 17-19, 2019, Juan Julián Merelo Guervós, Jonathan M. Garibaldi, Alejandro Linares-Barranco, Kurosh Madani, and Kevin Warwick (Eds.). ScitePress, 82--92. https://doi.org/10.5220/0008070100820092
[29]
Roman Kalkreuth. 2022. Towards Phenotypic Duplication and Inversion in Cartesian Genetic Programming. In Proceedings of the 14th International Joint Conference on Computational Intelligence, IJCCI 2022, Valletta, Malta, October 24-26, 2022, Thomas Bäck, Bas van Stein, Christian Wagner, Jonathan M. Garibaldi, H. K. Lam, Marie Cottrell, Faiyaz Doctor, Joaquim Filipe, Kevin Warwick, and Janusz Kacprzyk (Eds.). SCITEPRESS, 50--61. https://doi.org/10.5220/0011551000003332
[30]
Paul Kaufmann and Roman Kalkreuth. 2017. Parametrizing Cartesian Genetic Programming: An Empirical Study. In KI 2017: Advances in Artificial Intelligence - 40th Annual German Conference on AI, Dortmund, Germany, September 25-29, 2017, Proceedings (Lecture Notes in Computer Science, Vol. 10505), Gabriele Kern-Isberner, Johannes Fürnkranz, and Matthias Thimm (Eds.). Springer, 316--322. https://doi.org/10.1007/978-3-319-67190-1_26
[31]
John R. Koza. 1989. Hierarchical Genetic Algorithms Operating on Populations of Computer Programs. In Proceedings of the 11th International Joint Conference on Artificial Intelligence. Detroit, MI, USA, August 1989, N. S. Sridharan (Ed.). Morgan Kaufmann, 768--774. http://ijcai.org/Proceedings/89-1/Papers/123.pdf
[32]
John R. Koza. 1990. Concept Formation and Decision Tree Induction Using the Genetic Programming Paradigm. In Parallel Problem Solving from Nature, 1st Workshop, PPSN I, Dortmund, Germany, October 1-3, 1990, Proceedings (Lecture Notes in Computer Science, Vol. 496), Hans-Paul Schwefel and Reinhard Männer (Eds.). Springer, 124--128. https://doi.org/10.1007/BFb0029742
[33]
John R. Koza. 1990. Genetic Programming: A Paradigm for Genetically Breeding Populations of Computer Programs to Solve Problems. Technical Report. Stanford, CA, USA.
[34]
John R. Koza. 1990. A Hierarchical Approach to Learning the Boolean Multiplexer Function. In Proceedings of the First Workshop on Foundations of Genetic Algorithms. Bloomington Campus, Indiana, USA, July 15-18 1990, Gregory J. E. Rawlins (Ed.). Morgan Kaufmann, 171--192. https://doi.org/10.1016/b978-0-08-050684-5.50014-8
[35]
John R. Koza. 1993. Genetic programming - on the programming of computers by means of natural selection. MIT Press.
[36]
Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. 2016. The irace package: Iterated Racing for Automatic Algorithm Configuration. Operations Research Perspectives 3 (2016), 43--58. https://doi.org/10.1016/j.orp.2016.09.002
[37]
H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50--60. https://doi.org/10.1214/aoms/1177730491
[38]
James McDermott, Gabriel Kronberger, Patryk Orzechowski, Leonardo Vanneschi, Luca Manzoni, Roman Kalkreuth, and Mauro Castelli. 2022. Genetic Programming Benchmarks: Looking Back and Looking Forward. SIGEVOlution 15, 3, Article 1 (dec 2022), 19 pages. https://doi.org/10.1145/3578482.3578483
[39]
James McDermott, David Robert White, Sean Luke, Luca Manzoni, Mauro Castelli, Leonardo Vanneschi, Wojciech Jaskowski, Krzysztof Krawiec, Robin Harper, Kenneth A. De Jong, and Una-May O'Reilly. 2012. Genetic programming needs better benchmarks. In Genetic and Evolutionary Computation Conference, GECCO 12, Philadelphia, PA, USA, July 7-11, 2012, Terence Soule and Jason H. Moore (Eds.). ACM, 791--798. https://doi.org/10.1145/2330163.2330273
[40]
Julian F. Miller. 1999. An empirical study of the efficiency of learning boolean functions using a Cartesian Genetic Programming approach. In Proceedings of the Genetic and Evolutionary Computation Conference, Wolfgang Banzhaf, Jason Daida, Agoston E. Eiben, Max H. Garzon, Vasant Honavar, Mark Jakiela, and Robert E. Smith (Eds.), Vol. 2. Morgan Kaufmann, Orlando, Florida, USA, 1135--1142. http://citeseer.ist.psu.edu/153431.html
[41]
Julian F. Miller. 2011. Cartesian Genetic Programming. Springer Berlin Heidelberg, Berlin, Heidelberg, 17--34. https://doi.org/10.1007/978-3-642-17310-3_2
[42]
Julian F. Miller and Peter Thomson. 2000. Cartesian Genetic Programming. In Genetic Programming, European Conference, Edinburgh, Scotland, UK, April 15-16, 2000, Proceedings (Lecture Notes in Computer Science, Vol. 1802), Riccardo Poli, Wolfgang Banzhaf, William B. Langdon, Julian F. Miller, Peter Nordin, and Terence C. Fogarty (Eds.). Springer, 121--132. https://doi.org/10.1007/978-3-540-46239-2_9
[43]
J. F. Miller, P. Thomson, and T. Fogarty. 1997. Designing Electronic Circuits Using Evolutionary Algorithms. Arithmetic Circuits: A Case Study. In Genetic Algorithms and Evolution Strategies in Engineering and Computer Science. Wiley, 105--131.
[44]
Aneta Neumann, Frank Neumann, and Chao Qian. 2022. Evolutionary sub-modular optimisation: tutorial. In GECCO 22: Genetic and Evolutionary Computation Conference, Companion Volume, Boston, Massachusetts, USA, July 9-13, 2022, Jonathan E. Fieldsend and Markus Wagner (Eds.). ACM, 1427--1449. https://doi.org/10.1145/3520304.3533624
[45]
Patryk Orzechowski, William La Cava, and Jason H. Moore. 2018. Where Are We Now? A Large Benchmark Study of Recent Symbolic Regression Methods. In Proceedings of the Genetic and Evolutionary Computation Conference (Kyoto, Japan) (GECCO '18). Association for Computing Machinery, New York, NY, USA, 1183--1190. https://doi.org/10.1145/3205455.3205539
[46]
Patryk Orzechowski and Jason H. Moore. 2021. Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers. CoRR abs/2107.06475 (2021). arXiv:2107.06475 https://arxiv.org/abs/2107.06475
[47]
Stjepan Picek, Claude Carlet, Sylvain Guilley, Julian F Miller, and Domagoj Jakobovic. 2016. Evolutionary algorithms for boolean functions in diverse domains of cryptography. Evolutionary computation 24, 4 (2016), 667--694.
[48]
Stjepan Picek, Domagoj Jakobovic, Julian F Miller, Lejla Batina, and Marko Cupic. 2016. Cryptographic Boolean functions: One output, many design criteria. Applied Soft Computing 40 (2016), 635--653.
[49]
Riccardo Poli. 1997. Evolution of Graph-Like Programs with Parallel Distributed Genetic Programming. In Proceedings of the 7th International Conference on Genetic Algorithms, East Lansing, MI, USA, July 19-23, 1997, Thomas Bäck (Ed.). Morgan Kaufmann, 346--353.
[50]
Lukás Sekanina. 2012. Evolvable Hardware. In Handbook of Natural Computing, Grzegorz Rozenberg, Thomas Bäck, and Joost N. Kok (Eds.). Springer, 1657--1705. https://doi.org/10.1007/978-3-540-92910-9_50
[51]
Léo Françoso Dal Piccol Sotto, Paul Kaufmann, Timothy Atkinson, Roman Kalkreuth, and Márcio Porto Basgalupp. 2021. Graph representations in genetic programming. Genet. Program. Evolvable Mach. 22, 4 (2021), 607--636. https://doi.org/10.1007/s10710-021-09413-9
[52]
Xiaohu Tang, Deng Tang, Xiangyong Zeng, and Lei Hu. 2010. Balanced Boolean Functions with (Almost) Optimal Algebraic Immunity and Very High Nonlinearity. IACR Cryptology ePrint Archive 2010 (2010), 443.
[53]
Natalia Tokareva. 2011. On the number of bent functions from iterative constructions: lower bounds and hypotheses. Cryptology ePrint Archive (2011).
[54]
Konstantinos Varelas, Ouassim Ait ElHara, Dimo Brockhoff, Nikolaus Hansen, Duc Manh Nguyen, Tea Tusar, and Anne Auger. 2020. Benchmarking large-scale continuous optimizers: The bbob-largescale testbed, a COCO software guide and beyond. Appl. Soft Comput. 97, Part (2020), 106737. https://doi.org/10.1016/j.asoc.2020.106737
[55]
Zdenek Vasicek. 2017. Bridging the Gap Between Evolvable Hardware and Industry Using Cartesian Genetic Programming. In Inspired by Nature: Essays Presented to Julian F. Miller on the Occasion of his 60th Birthday, Susan Stepney and Andrew Adamatzky (Eds.). Emergence, Complexity and Computation, Vol. 28. Springer, Chapter 2, 39--55. https://doi.org/
[56]
Zdenek Vasicek and Lukas Sekanina. 2014. How to evolve complex combinational circuits from scratch?. In 2014 IEEE International Conference on Evolvable Systems, ICES 2014, Orlando, FL, USA, December 9-12, 2014. IEEE, 133--140. https://doi.org/10.1109/ICES.2014.7008732
[57]
Zdenek Vasicek and Karel Slany. 2012. Efficient Phenotype Evaluation in Cartesian Genetic Programming. In Lecture Notes in Computer Science. Springer Berlin Heidelberg, 266--278. https://doi.org/10.1007/978-3-642-29139-5_23
[58]
James Alfred Walker, Katharina Völk, Stephen L. Smith, and Julian Francis Miller. 2009. Parallel evolution using multi-chromosome cartesian genetic programming. Genet. Program. Evolvable Mach. 10, 4 (2009), 417--445. https://doi.org/10.1007/s10710-009-9093-2
[59]
Hao Wang, Diederick Vermetten, Furong Ye, Carola Doerr, and Thomas Bäck. 2022. IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics. ACM Trans. Evol. Learn. Optim. 2, 1 (2022), 3:1--3:29. https://doi.org/10.1145/3510426
[60]
Ingo Wegener. 1991. The Complexity of Boolean Functions (Wiley Teubner on Applicable Theory in Computer Science). (1991).
[61]
David Robert White, James McDermott, Mauro Castelli, Luca Manzoni, Brian W. Goldman, Gabriel Kronberger, Wojciech Jaskowski, Una-May O'Reilly, and Sean Luke. 2013. Better GP benchmarks: community survey results and proposals. Genet. Program. Evolvable Mach. 14, 1 (2013), 3--29. https://doi.org/10.1007/s10710-012-9177-2
[62]
Stewart W. Wilson. 1987. Classifier Systems and the Animat Problem. Mach. Learn. 2, 3 (1987), 199--228. https://doi.org/10.1007/BF00058679
[63]
S. Yang. 1991. Logic Synthesis and Optimization Benchmarks User Guide: Version 3.0. Technical Report. MCNC Technical Report.
[64]
Furong Ye, Hao Wang, Carola Doerr, and Thomas Bäck. 2020. Benchmarking a (μ + λ) Genetic Algorithm with Configurable Crossover Probability. In Parallel Problem Solving from Nature - PPSN XVI- 16th International Conference, PPSN 2020, Leiden, The Netherlands, September 5-9, 2020, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 12270), Thomas Bäck, Mike Preuss, André H. Deutz, Hao Wang, Carola Doerr, Michael T. M. Emmerich, and Heike Trautmann (Eds.). Springer, 699--713. https://doi.org/10.1007/978-3-030-58115-2_49
[65]
Tina Yu and Julian Miller. 2002. Finding Needles in Haystacks Is Not Hard with Neutrality. In Genetic Programming, James A. Foster, Evelyne Lutton, Julian Miller, Conor Ryan, and Andrea Tettamanzi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 13--25.

Cited By

View all
  • (2024)CGP++ : A Modern C++ Implementation of Cartesian Genetic ProgrammingProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654092(13-22)Online publication date: 14-Jul-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
FOGA '23: Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms
August 2023
169 pages
ISBN:9798400702020
DOI:10.1145/3594805
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2023

Check for updates

Author Tags

  1. Benchmarking
  2. Genetic Programming
  3. Logic Synthesis

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

FOGA '23
Sponsor:
FOGA '23: Foundations of Genetic Algorithms XVII
August 30 - September 1, 2023
Potsdam, Germany

Acceptance Rates

Overall Acceptance Rate 72 of 131 submissions, 55%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)297
  • Downloads (Last 6 weeks)45
Reflects downloads up to 02 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)CGP++ : A Modern C++ Implementation of Cartesian Genetic ProgrammingProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654092(13-22)Online publication date: 14-Jul-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media