research-article

Maximum number of generations as a stopping criterion considered harmful

Authors:

Matej ČrepinšekAuthors Info & Claims

Volume 128, Issue C

https://doi.org/10.1016/j.asoc.2022.109478

Published: 01 October 2022 Publication History

Abstract

Evolutionary algorithms have been shown to be very effective in solving complex optimization problems. This has driven the research community in the development of novel, even more efficient evolutionary algorithms. The newly proposed algorithms need to be evaluated and compared with existing state-of-the-art algorithms, usually by employing benchmarks. However, comparing evolutionary algorithms is a complicated task, which involves many factors that must be considered to ensure a fair and unbiased comparison. In this paper, we focus on the impact of stopping criteria in the comparison process. Their job is to stop the algorithms in such a way that each algorithm has a fair opportunity to solve the problem. Although they are not given much attention, they play a vital role in the comparison process. In the paper, we compared different stopping criteria with different settings, to show their impact on the comparison results. The results show that stopping criteria play a vital role in the comparison, as they can produce statistically significant differences in the rankings of evolutionary algorithms. The experiments have shown that in one case an algorithm consumed 50 times more evaluations in a single generation, giving it a considerable advantage when max gen was used as the stopping criterion, which puts the validity of most published work in question.

Highlights

•

Overview of four existing stopping criteria used in benchmarking.

•

Comparison of seven evolutionary algorithms by using a chess rating system.

•

Two benchmarks consisting of 36 standard problems and a real-world problem.

•

Presented guidelines for applying stopping criteria in benchmarking.

References

[1]

Bartz-Beielstein T., Doerr C., Berg D.v.d., Bossek J., Chandrasekaran S., Eftimov T., Fischbach A., Kerschke P., La Cava W., Lopez-Ibanez M., et al., Benchmarking in optimization: Best practice and open issues, 2020, arXiv preprint arXiv:2007.03488.

[2]

Barr R.S., Golden B.L., Kelly J.P., Resende M.G., Stewart W.R., Designing and reporting on computational experiments with heuristic methods, J. Heuristics 1 (1) (1995) 9–32.

Digital Library

[3]

Eiben A.E., Jelasity M., A critical note on experimental research methodology in EC, in: Proceedings of the 2002 Congress on Evolutionary Computation. CEC’02 (Cat. No. 02TH8600), Vol. 1, IEEE, 2002, pp. 582–587.

[4]

LaTorre A., Molina D., Osaba E., Del Ser J., Herrera F., Fairness in bio-inspired optimization research: A prescription of methodological guidelines for comparing meta-heuristics, 2020, arXiv preprint arXiv:2004.09969.

[5]

Črepinšek M., Liu S.-H., Mernik M., Replication and comparison of computational experiments in applied evolutionary computing: common pitfalls and guidelines to avoid them, Appl. Soft Comput. 19 (2014) 161–170.

[6]

Ravber M., Mernik M., Črepinšek M., The impact of quality indicators on the rating of multi-objective evolutionary algorithms, in: Conference on Bioinspired Optimization Methods and their Applications (BIOMA 2016), 2016, pp. 119–130.

[7]

Y. Liu, A. Zhou, H. Zhang, Termination detection strategies in evolutionary algorithms: a survey, in: Proceedings of the Genetic and Evolutionary Computation Conference, 2018, pp. 1063–1070.

[8]

Ghoreishi S.N., Clausen A., Joergensen B.N., Termination criteria in evolutionary algorithms: A survey, in: IJCCI, 2017, pp. 373–384.

[9]

Kazikova A., Pluhacek M., Senkerik R., How does the number of objective function evaluations impact our understanding of metaheuristics behavior?, IEEE Access 9 (2021) 44032–44048.

[10]

Molina D., Poyatos J., Del Ser J., García S., Hussain A., Herrera F., Comprehensive taxonomies of nature-and bio-inspired optimization: Inspiration versus algorithmic behavior, critical analysis recommendations, Cogn. Comput. 12 (5) (2020) 897–939.

[11]

Révész P., Random Walk in Random and Non-Random Environments, World Scientific, 2013.

[12]

Črepinšek M., Liu S.-H., Mernik L., Mernik M., Is a comparison of results meaningful from the inexact replications of computational experiments?, Soft Comput. 20 (1) (2016) 223–235.

[13]

Castelli M., Manzoni L., Mariot L., Nobile M.S., Tangherloni A., Salp swarm optimization: A critical review, Expert Syst. Appl. 189 (2022).

[14]

Tanabe R., Fukunaga A.S., Improving the search performance of SHADE using linear population size reduction, in: 2014 IEEE Congress on Evolutionary Computation (CEC), IEEE, 2014, pp. 1658–1665.

[15]

Mirjalili S., Mirjalili S.M., Lewis A., Grey wolf optimizer, Adv. Eng. Softw. 69 (2014) 46–61.

Digital Library

[16]

Mernik M., Liu S.-H., Karaboga D., Črepinšek M., On clarifying misconceptions when comparing variants of the artificial bee colony algorithm by offering a new implementation, Inform. Sci. 291 (2015) 115–127.

Digital Library

[17]

Črepinšek M., Liu S.-H., Mernik L., A note on teaching–learning-based optimization algorithm, Inform. Sci. 212 (2012) 79–93.

[18]

Zhang J., Sanderson A.C., JADE: adaptive differential evolution with optional external archive, IEEE Trans. Evol. Comput. 13 (5) (2009) 945–958.

Digital Library

[19]

Gonzalez-Fernandez Y., Chen S., Leaders and followers—a new metaheuristic to avoid the bias of accumulated information, in: 2015 IEEE Congress on Evolutionary Computation (CEC), IEEE, 2015, pp. 776–783.

[20]

Karaboga D., Basturk B., A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm, J. Global Optim. 39 (3) (2007) 459–471.

Digital Library

[21]

Rao R.V., Savsani V.J., Vakharia D., Teaching–learning-based optimization: a novel method for constrained mechanical design optimization problems, Comput. Aided Des. 43 (3) (2011) 303–315.

Digital Library

[22]

Zelinka I., SOMA—self-organizing migrating algorithm, in: Self-Organizing Migrating Algorithm, Springer, 2016, pp. 3–49.

[23]

J. Kennedy, R. Eberhart, Particle swarm optimization, in: Proceedings of ICNN’95 - International Conference on Neural Networks, Vol. 4, 1995, pp. 1942–1948.

[24]

Geem Z.W., Kim J.H., Loganathan G.V., A new heuristic optimization algorithm: harmony search, Simulation 76 (2) (2001) 60–68.

[25]

EARS - Evolutionary algorithms rating system (Github), 2019, available at https://github.com/UM-LPM/EARS. URL https://github.com/UM-LPM/EARS.

[26]

Veček N., Mernik M., Črepinšek M., A chess rating system for evolutionary algorithms: a new method for the comparison and ranking of evolutionary algorithms, Inform. Sci. 277 (2014) 656–679.

[27]

B.J. Jain, H. Pohlheim, J. Wegener, On termination criteria of evolutionary algorithms, in: Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation, 2001, pp. 768–768.

[28]

Engelbrecht A.P., Fitness function evaluations: A fair stopping condition?, in: 2014 IEEE Symposium on Swarm Intelligence, IEEE, 2014, pp. 1–8.

[29]

Črepinšek M., Ravber M., Mernik M., Kosar T., Tuning multi-objective evolutionary algorithms on different sized problem sets, Mathematics 7 (9) (2019) 824.

[30]

Jerebic J., Mernik M., Liu S.-H., Ravber M., Baketarić M., Mernik L., Črepinšek M., A novel direct measure of exploration and exploitation based on attraction basins, Expert Syst. Appl. 167 (2021).

[31]

Baketarić M., Mernik M., Kosar T., Attraction basins in metaheuristics: A systematic mapping study, Mathematics 9 (23) (2021) 3036.

[32]

Zielinski K., Weitkemper P., Laur R., Kammeyer K.-D., Examination of stopping criteria for differential evolution based on a power allocation problem, J. Electr. Eng. 7 (3) (2007) 8.

[33]

Zielinski K., Peters D., Laur R., Stopping criteria for single-objective optimization, 2005.

[34]

Aytug H., Koehler G.J., New stopping criterion for genetic algorithms, European J. Oper. Res. 126 (3) (2000) 662–674.

[35]

Kim J.-L., Genetic algorithm stopping criteria for optimization of construction resource scheduling problems, Constr. Manag. Econ. 31 (1) (2013) 3–19.

[36]

Wessing S., Preuss M., Trautmann H., Stopping criteria for multimodal optimization, in: International Conference on Parallel Problem Solving from Nature, Springer, 2014, pp. 141–150.

[37]

Gil D., Roche D., Borràs A., Giraldo J., Terminating evolutionary algorithms at their steady state, Comput. Optim. Appl. 61 (2) (2015) 489–515.

[38]

Rhinehart R.R., Convergence criterion in optimization of stochastic processes, Comput. Chem. Eng. 68 (2014) 1–6.

[39]

Hermadi I., Lokan C., Sarker R., Dynamic stopping criteria for search-based test data generation for path testing, Inf. Softw. Technol. 56 (4) (2014) 395–407.

[40]

Cohen Y., Gelbard R., Amorim M., When to stop? A new stop criterion for combinatorial optimization search techniques, in: Engineering Digital Transformation, Springer, 2019, pp. 211–218.

[41]

Zielinski K., Laur R., Stopping criteria for differential evolution in constrained single-objective optimization, in: Advances in Differential Evolution, Springer, 2008, pp. 111–138.

[42]

Beiranvand V., Hare W., Lucet Y., Best practices for comparing optimization algorithms, Opt. Eng. 18 (4) (2017) 815–848.

[43]

Jesenik M., Mernik M., Črepinšek M., Ravber M., Trlep M., Searching for soil models’ parameters using metaheuristics, Appl. Soft Comput. 69 (2018) 131–148.

[44]

Rao R.V., Savsani V.J., Vakharia D., Teaching–learning-based optimization: an optimization method for continuous non-linear large scale problems, Inform. Sci. 183 (1) (2012) 1–15.

Digital Library

[45]

Tien J.-P., Li T.-H.S., Hybrid Taguchi-chaos of multilevel immune and the artificial bee colony algorithm for parameter identification of chaotic systems, Comput. Math. Appl. 64 (5) (2012) 1108–1119.

[46]

Hsieh T.-J., Hsiao H.-F., Yeh W.-C., Mining financial distress trend data using penalty guided support vector machines based on hybrid of particle swarm optimization and artificial bee colony algorithm, Neurocomputing 82 (2012) 196–206.

[47]

Li G., Niu P., Xiao X., Development and investigation of efficient artificial bee colony algorithm for numerical function optimization, Appl. Soft Comput. 12 (1) (2012) 320–332.

Digital Library

[48]

Allawi Z.T., Ibraheem I.K., Humaidi A.J., Fine-tuning meta-heuristic algorithm for global optimization, Processes 7 (10) (2019) 657.

[49]

Meng Z., Pan J.-S., Monkey king evolution: a new memetic evolutionary algorithm and its application in vehicle fuel consumption optimization, Knowl.-Based Syst. 97 (2016) 144–157.

[50]

Mousavirad S.J., Ebrahimpour-Komleh H., Human mental search: a new population-based metaheuristic optimization algorithm, Appl. Intell. 47 (3) (2017) 850–887.

Digital Library

[51]

Seif Z., Ahmadi M.B., An opposition-based algorithm for function optimization, Eng. Appl. Artif. Intell. 37 (2015) 293–306.

[52]

Panda G., Pradhan P.M., Majhi B., IIR system identification using cat swarm optimization, Expert Syst. Appl. 38 (10) (2011) 12671–12683.

[53]

Meng A.-b., Chen Y.-c., Yin H., Chen S.-z., Crisscross optimization algorithm and its application, Knowl.-Based Syst. 67 (2014) 218–229.

Digital Library

[54]

Tsai H.-C., Roach infestation optimization with friendship centers, Eng. Appl. Artif. Intell. 39 (2015) 109–119.

[55]

Mirjalili S., The ant lion optimizer, Adv. Eng. Softw. 83 (2015) 80–98.

Digital Library

[56]

Neshat M., Sepidnam G., Sargolzaei M., Toosi A.N., Artificial fish swarm algorithm: a survey of the state-of-the-art, hybridization, combinatorial and indicative applications, Artif. Intell. Rev. 42 (4) (2014) 965–997.

Digital Library

[57]

Li X., Yin M., Modified cuckoo search algorithm with self adaptive parameter method, Inform. Sci. 298 (2015) 80–97.

Digital Library

[58]

Ghasemi M., Ghavidel S., Rahmani S., Roosta A., Falah H., A novel hybrid algorithm of imperialist competitive algorithm and teaching learning algorithm for optimal power flow problem with non-smooth cost functions, Eng. Appl. Artif. Intell. 29 (2014) 54–69.

Digital Library

[59]

Salimi H., Stochastic fractal search: a powerful metaheuristic algorithm, Knowl.-Based Syst. 75 (2015) 1–18.

Digital Library

[60]

Yap D.F., Koh S., Tiong S.K., Prajindra S., A hybrid artificial immune systems for multimodal function optimization and its application in engineering problem, Artif. Intell. Rev. 38 (4) (2012) 291–301.

[61]

Kuo R.-J., Zulvia F.E., The gradient evolution algorithm: A new metaheuristic, Inform. Sci. 316 (2015) 246–265.

[62]

Beheshti Z., Shamsuddin S.M., Hasan S., Memetic binary particle swarm optimization for discrete optimization problems, Inform. Sci. 299 (2015) 58–84.

[63]

Zheng L., An improved monkey algorithm with dynamic adaptation, Appl. Math. Comput. 222 (2013) 645–657.

[64]

Huang J., Gao L., Li X., An effective teaching-learning-based cuckoo search algorithm for parameter optimization problems in structure designing and machining processes, Appl. Soft Comput. 36 (2015) 349–356.

Digital Library

[65]

Long N.C., Meesad P., Unger H., A highly accurate firefly based algorithm for heart disease prediction, Expert Syst. Appl. 42 (21) (2015) 8221–8231.

[66]

Yang Y., Shi Y., Xia S., Advanced discussion mechanism-based brain storm optimization algorithm, Soft Comput. 19 (10) (2015) 2997–3007.

[67]

Chen J., Yang Q., Ni J., Xie Y., Cheng S., An improved fireworks algorithm with landscape information for balancing exploration and exploitation, in: 2015 IEEE Congress on Evolutionary Computation (CEC), IEEE, 2015, pp. 1272–1279.

[68]

Su C.-T., Lin H.-C., Applying electromagnetism-like mechanism for feature selection, Inform. Sci. 181 (5) (2011) 972–986.

[69]

Shareef H., Ibrahim A.A., Mutlag A.H., Lightning search algorithm, Appl. Soft Comput. 36 (2015) 315–333.

Digital Library

[70]

Hellwig M., Beyer H.-G., Benchmarking evolutionary algorithms for single objective real-valued constrained optimization–a critical review, Swarm Evol. Comput. 44 (2019) 927–944.

[71]

Tušar T., Gantar K., Koblar V., Ženko B., Filipič B., A study of overfitting in optimization of a manufacturing quality control procedure, Appl. Soft Comput. 59 (2017) 77–87.

[72]

Glorieux E., Constructive Cooperative Coevolution for Optimising Interacting Production Stations, (Ph.D. thesis) University West, 2015.

[73]

Silberholz J., Golden B., Comparison of metaheuristics, in: Handbook of Metaheuristics, Springer, 2010, pp. 625–640.

[74]

Balande U., Shrimankar D., SRIFA: Stochastic ranking with improved-firefly-algorithm for constrained optimization engineering design problems, Mathematics 7 (3) (2019) 250.

[75]

Cassola M., Talagorria A., Pardo A., Viera M., A gradual type system for elixir, J. Comput. Lang. (2021).

[76]

López-Fraguas F.J., Montenegro M., Suárez-García G., Deriving overloaded success type schemes in Erlang, J. Comput. Lang. 58 (2020).

[77]

Pieper R., Löff J., Hoffmann R.B., Griebler D., Fernandes L.G., High-level and efficient structured stream parallelism for rust on multi-cores, J. Comput. Lang. 65 (2021).

[78]

Hansen N., Auger A., Ros R., Mersmann O., Tušar T., Brockhoff D., COCO: A platform for comparing continuous optimizers in a black-box setting, Optim. Methods Softw. 36 (1) (2021) 114–144.

[79]

Tian Y., Cheng R., Zhang X., Jin Y., PlatEMO: A MATLAB platform for evolutionary multi-objective optimization, IEEE Comput. Intell. Mag. 12 (4) (2017) 73–87.

Digital Library

[80]

Fortin F.-A., De Rainville F.-M., Gardner M.-A., Parizeau M., Gagné C., DEAP: Evolutionary algorithms made easy, J. Mach. Learn. Res. 13 (2012) 2171–2175.

Digital Library

[81]

Durillo J.J., Nebro A.J., JMetal: A Java framework for multi-objective optimization, Adv. Eng. Softw. 42 (10) (2011) 760–771.

Digital Library

[82]

Biswas A., Biswas B., Kumar A., Mishra K., Particle swarm optimisation with time varying cognitive avoidance component, Int. J. Comput. Sci. Eng. 16 (1) (2018) 27–41.

[83]

Atabaki M.S., Mohammadi M., Naderi B., Hybrid genetic algorithm and invasive weed optimization via priority based encoding for location-allocation decisions in a three-stage supply chain, Asia-Pac. J. Oper. Res. 34 (02) (2017).

[84]

Atabaki M.S., Khamseh A.A., Mohammadi M., A priority-based firefly algorithm for network design of a closed-loop supply chain with price-sensitive demand, Comput. Ind. Eng. 135 (2019) 814–837.

[85]

Jahani E., Chizari M., Tackling global optimization problems with a novel algorithm–Mouth Brooding Fish algorithm, Appl. Soft Comput. 62 (2018) 987–1002.

[86]

Jesenik M., Beković M., Hamler A., Trlep M., Analytical modelling of a magnetization curve obtained by the measurements of magnetic materials’ properties using evolutionary algorithms, Appl. Soft Comput. 52 (2017) 387–408.

[87]

Veček N., Črepinšek M., Mernik M., On the influence of the number of algorithms, problems, and independent runs in the comparison of evolutionary algorithms, Appl. Soft Comput. 54 (2017) 23–45.

[88]

Veček N., Liu S.-H., Črepinšek M., Mernik M., On the importance of the artificial bee colony control parameter ‘limit, Inf. Technol. Control 46 (4) (2017) 566–604.

[89]

Ravber M., Mernik M., Črepinšek M., Ranking multi-objective evolutionary algorithms using a chess rating system with quality indicator ensemble, in: IEEE Congress on Evolutionary Computation (CEC 2017), IEEE, 2017, pp. 1503–1510.

[90]

Ravber M., Črepinšek M., Mernik M., Kosar T., Tuning multi-objective optimization algorithms for the integration and testing order problem, in: International Conference on Bioinspired Methods and their Applications, Springer, 2018, pp. 234–245.

[91]

Ravber M., Kovačević Ž., Črepinšek M., Mernik M., Inferring absolutely non-circular attribute grammars with a memetic algorithm, Appl. Soft Comput. 100 (2021).

[92]

Glickman M.E., Example of the Glicko-2 System, Boston University, 2012.

[93]

Ezugwu A.E., Adeleke O.J., Akinyelu A.A., Viriri S., A conceptual comparison of several metaheuristic algorithms on continuous optimisation problems, Neural Comput. Appl. 32 (10) (2020) 6207–6251.

Digital Library

[94]

Hansen N., Tusar T., Mersmann O., Auger A., Brockhoff D., COCO: The experimental procedure, 2016, arXiv preprint arXiv:1603.08776.

[95]

Weyland D., A critical analysis of the harmony search algorithm—How not to solve sudoku, Oper. Res. Perspect. 2 (2015) 97–105.

[96]

Tangherloni A., Spolaor S., Cazzaniga P., Besozzi D., Rundo L., Mauri G., Nobile M.S., Biochemical parameter estimation vs. benchmark functions: A comparative study of optimization performance and representation design, Appl. Soft Comput. 81 (2019).

Digital Library

[97]

Gonos I.F., Stathopulos I.A., Estimation of multilayer soil parameters using genetic algorithms, IEEE Trans. Power Deliv. 20 (1) (2005) 100–106.

[98]

Črepinšek M., Liu S.-H., Mernik M., Ravber M., Long term memory assistance for evolutionary algorithms, Mathematics 7 (11) (2019) 1129.

[99]

Stewart D.B., Measuring execution time and real-time performance, in: Embedded Systems Conference (ESC), Vol. 141, Citeseer, 2001.

[100]

Yuan X., Wang L., Yuan Y., Zhang Y., Cao B., Yang B., A modified differential evolution approach for dynamic economic dispatch with valve-point effects, Energy Convers. Manage. 49 (12) (2008) 3447–3453.

[101]

Chen Q., Liu B., Zhang Q., Liang J., Suganthan P., Qu B., Problem Definitions and Evaluation Criteria for CEC 2015 Special Session on Bound Constrained Single-Objective Computationally Expensive Numerical Optimization, Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou, China and Technical Report, Nanyang Technological University, 2014.

[102]

Dijkstra E.W., Go to statement considered harmful, Commun. ACM 11 (3) (1968) 147–148.

[103]

Rapin J., Teytaud O., Nevergrad - A gradient-free optimization platform, 2018, https://GitHub.com/FacebookResearch/Nevergrad.

[104]

Wang H., Vermettern D., Ye F., Doerr C., Bäck T., IOHanalyzer: Performance analysis for iterative optimization heuristic, 2020, arXiv e-prints:2007.03953. arXiv:2007.03953.

Cited By

Song LDong YGuo QMeng YZhao G(2023)An adaptive differential evolution algorithm with DBSCAN for the integrated slab allocation problem in steel industryApplied Soft Computing10.1016/j.asoc.2023.110665146:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.asoc.2023.110665

Index Terms

Maximum number of generations as a stopping criterion considered harmful
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies
      1. Heuristic function construction
2. Theory of computation
  1. Design and analysis of algorithms

Index terms have been assigned to the content through auto-classification.

Recommendations

A Variance-Value-Based Stopping Criterion for BICM-ID
ISIC '11: Proceedings of the 2011 International Conference on Information Security and Intelligence Control

Stopping criterion is an efficient technique to reduce the unnecessary iterations of the iterative decoding process. So far, the overwhelming majority of the existing stopping criteria are designed for turbo-decoder, not for bit-interleaved coded ...
Enhanced Cross-Entropy Based Stopping Criteria at Low Signal-to-Noise Ratio Regions
ICCAE 2018: Proceedings of the 2018 10th International Conference on Computer and Automation Engineering

This paper presents an enhancement of Cross-Entropy (CE)-based stopping criteria at a low signal-to-noise ratios (SNRs) region. The existing CE-based stopping criteria fail to stop the iteration and cause the maximum iteration at low SNRs. Intuitively, ...
Towards a self-stopping evolutionary algorithm using coupling from the past
GECCO '05: Proceedings of the 7th annual conference on Genetic and evolutionary computation

In this paper a stopping criterion for a particular class of evolutionary algorithms is devised. First, a model of a generic evolutionary algorithm using iterated random maps is presented. The model allows the exploration of a connection between ...

Comments

Information & Contributors

Information

Published In

cover image Applied Soft Computing

Applied Soft Computing Volume 128, Issue C

Oct 2022

902 pages

ISSN:1568-4946

Issue’s Table of Contents

The Author(s).

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 October 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Song LDong YGuo QMeng YZhao G(2023)An adaptive differential evolution algorithm with DBSCAN for the integrated slab allocation problem in steel industryApplied Soft Computing10.1016/j.asoc.2023.110665146:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.asoc.2023.110665

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents