research-article

Computing Star Discrepancies with Numerical Black-Box Optimization Algorithms

Authors:

François Clément,

Diederick Vermetten,

Jacob De Nobel,

Alexandre D. Jesus,

Carola DoerrAuthors Info & Claims

GECCO '23: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 1330 - 1338

https://doi.org/10.1145/3583131.3590456

Published: 12 July 2023 Publication History

Abstract

The L_∞ star discrepancy is a measure for the regularity of a finite set of points taken from [0, 1)^d. Low discrepancy point sets are highly relevant for Quasi-Monte Carlo methods in numerical integration and several other applications. Unfortunately, computing the L_∞ star discrepancy of a given point set is known to be a hard problem, with the best exact algorithms falling short for even moderate dimensions around 8. However, despite the difficulty of finding the global maximum that defines the L_∞ star discrepancy of the set, local evaluations at selected points are inexpensive. This makes the problem tractable by black-box optimization approaches.

In this work we compare 8 popular numerical black-box optimization algorithms on the L_∞ star discrepancy computation problem, using a wide set of instances in dimensions 2 to 15. We show that all used optimizers perform very badly on a large majority of the instances and that in many cases random search outperforms even the more sophisticated solvers. We suspect that state-of-the-art numerical black-box optimization techniques fail to capture the global structure of the problem, an important shortcoming that may guide their future development.

We also provide a parallel implementation of the best-known algorithm to compute the discrepancy.

References

[1]

E. Ayguade, N. Copty, A. Duran, J. Hoeflinger, Y. Lin, F. Massaioli, X. Teruel, P. Unnikrishnan, and G. Zhang. 2009. The Design of OpenMP Tasks. IEEE Transactions on Parallel and Distributed Systems 20, 3 (March 2009), 404--418.

Digital Library

[2]

J. Bossek, C. Doerr, P. Kerschke, A. Neumann, and F. Neumann. 2020. Evolving Sampling Strategies for One-Shot Optimization Tasks. In Proc. of Parallel Problem Solving from Nature (PPSN) (LNCS, Vol. 12269). Springer, 111--124.

Digital Library

[3]

O. Bousquet, S. Gelly, K. Kurach, O. Teytaud, and D. Vincent. 2017. Critical Hyper-Parameters: No Random, No Cry. CoRR abs/1706.03200 (2017). arXiv:1706.03200 http://arxiv.org/abs/1706.03200

[4]

K. I. Chatzilygeroudis, A. Cully, V. Vassiliades, and J.-B. Mouret. 2020. Quality-Diversity Optimization: a Novel Branch of Stochastic Optimization. CoRR abs/2012.04322 (2020). arXiv:2012.04322 https://arxiv.org/abs/2012.04322

[5]

F. Clément, C. Doerr, and L. Paquete. 2022. Star Discrepancy Subset Selection: Problem Formulation and Efficient Approaches for Low Dimensions. Journal of Complexity 70 (2022), 101645.

Digital Library

[6]

F. Clément, D. Vermetten, J. de Nobel, A. D. Jesus, L. Paquete, and C. Doerr. 2023. Reproducibility files and additional figures.

[7]

J. de Nobel, F. Ye, D. Vermetten, H. Wang, C. Doerr, and T. Bäck. 2021. IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics. CoRR abs/2111.04077 (2021). arXiv:2111.04077 https://arxiv.org/abs/2111.04077

[8]

J. Dick and F. Pillichshammer. 2010. Digital Nets and Sequences. Cambridge University Press.

[9]

D.P. Dobkin, D. Eppstein, and D.P. Mitchell. 1996. Computing the Discrepancy with Applications to Supersampling Patterns. ACM Trans. Graph. 15 (1996), 354--376.

Digital Library

[10]

C. Doerr, M. Gnewuch, and M. Wahlström. 2014. Calculation of Discrepancy Measures and Applications. In A Panorama of Discrepancy Theory, W. Chen, A. Srivastav, and G. Travaglini (Eds.). Springer, 621--678.

[11]

C. Doerr and F.-M. De Rainville. 2013. Constructing Low Star Discrepancy Point Sets with Genetic Algorithms. In Proc. of Genetic and Evolutionary Computation Conference (GECCO). ACM, 789--796.

Digital Library

[12]

C. Doerr, H. Wang, F. Ye, S. van Rijn, and T. Bäck. 2018. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. CoRR abs/1810.05281 (2018). http://arxiv.org/abs/1810.05281 Available at http://arxiv.org/abs/1810.05281. A more up-to-date documentation of IOHprofiler is available at https://iohprofiler.github.io/.

[13]

G. Dueck and T. Scheuer. 1990. Threshold accepting: a general purpose optimization algorithm appearing superior to simulated annealing. J. Comput. Phys. 90 (1990), 161--175.

Digital Library

[14]

B. Fox. 1986. Algorithm 647:Implementation and Relative Efficiency of Quasirandom Sequence Generators. ACM Trans. Math. Software 12, 4 (1986), 362--376.

Digital Library

[15]

S. Galanti and A. Jung. 1997. Low-discrepancy sequences: Monte-Carlo simulation of option prices. J. Deriv (1997), 63--83.

[16]

P. Giannopoulos, C. Knauer, M. Wahlström, and D. Werner. 2012. Hardness of discrepancy computation and ε-net verification in high dimension. J. Complex. 28 (2012), 162--176.

Digital Library

[17]

M. Gnewuch, A. Srivastav, and C. Winzen. 2009. Finding Optimal volume subintervals with k points and calculating the star discrepancy are NP-hard problems. J. Complex. 25 (2009), 115--127.

Digital Library

[18]

M. Gnewuch, M. Wahlström, and C. Winzen. 2012. A New Randomized Algorithm to Approximate the Star Discrepancy Based on Threshold Accepting. SIAM J. Numerical Analysis 50 (2012), 781--807.

Digital Library

[19]

J. H. Halton. 1960. On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals. Numer. Math. 2 (1960), 84--90.

Digital Library

[20]

N. Hansen and A. Ostermeier. 2001. Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation 9, 2 (2001), 159--195.

Digital Library

[21]

E. Hlawka. 1961. Funktionen von beschrankter Variation in der Theorie der Gleichverteilung. Ann. Mat. Pum Appl. 54 (1961), 325--333.

[22]

J. Kennedy and R. Eberhart. 1995. Particle swarm optimization. In Proc. of ICNN'95 - International Conference on Neural Networks, Vol. 4. 1942--1948.

[23]

S. Kirkpatrick, C.D. Gelatt, and M.P. Vecchi. 1983. Optimization by Simulated Annealing. Science 220 (1983), 671--680.

[24]

J. F. Koksma. 1942. Een algemeene stelling inuit de theorie der gelijkmatige verdeeling modulo 1. Mathematica (Zutphen) 11 (1942), 7--11.

[25]

P. Larrañaga and J.A. Lozano. 2001. Estimation of distribution algorithms: A new tool for evolutionary computation. Vol. 2. Springer Science & Business Media.

[26]

L. Meunier, H. Rakotoarison, P. K. Wong, B. Roziere, J. Rapin, O. Teytaud, A. Moreau, and C. Doerr. 2021. Black-box optimization revisited: Improving algorithm selection wizards through massive benchmarking. IEEE Transactions on Evolutionary Computation 26, 3 (2021), 490--500.

[27]

A. Neumann, W. Gao, C. Doerr, F. Neumann, and M. Wagner. 2018. Discrepancy-based evolutionary diversity optimization. In Proc. of Genetic and Evolutionary Computation Conference (GECCO). ACM, 991--998.

Digital Library

[28]

A. Neumann, W. Gao, M. Wagner, and F. Neumann. 2019. Evolutionary diversity optimization using multi-objective indicators. In Proc. of Genetic and Evolutionary Computation Conference (GECCO). ACM, 837--845.

Digital Library

[29]

H. Niederreiter. 1972. Discrepancy and Convex Programming. Ann. Mat. Pura Appl. 93 (1972), 89--97.

[30]

H. Niederreiter. 1992. Random Number Generation and Quasi-Monte Carlo Methods. Society for Industrial and Applied Mathematics.

Digital Library

[31]

L. Paulin, N. Bonneel, D. Coeurjoly, J.-C. Iehl, A. Keller, and V. Ostromoukhov. 2022. MatBuilder: Mastering Sampling Uniformity over projections. ACM Transactions on Graphics (proceedings of SIGGRAPH) (2022).

[32]

M. J. D. Powell. 1994. A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation. In Advances in Optimization and Numerical Analysis, S. Gomez and J.-P. Hennart (Eds.). Springer Netherlands, Dordrecht, 51--67.

[33]

J.K. Pugh, L.B. Soros, and K.O. Stanley. 2016. Quality Diversity: A New Frontier for Evolutionary Computation. Frontiers Robotics AI 3 (2016), 40.

[34]

J.K. Pugh, L.B. Soros, P.A. Szerlip, and K.O. Stanley. 2015. Confronting the Challenge of Quality Diversity. In Proc. of Genetic and Evolutionary Computation Conference (GECCO). ACM, 967--974.

Digital Library

[35]

J. Rapin and O. Teytaud. 2018. Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad.

[36]

T.J. Santner, B.J. Williams, and W.I. Notz. 2003. The Design and Analysis of Computer Experiments. Springer Series in Statistics, Springer.

[37]

O. Shir. 2012. Niching in Evolutionary Algorithms. In Handbook of Natural Computing, Grzegorz Rozenberg, Thomas Bäck, and Joost N. Kok (Eds.). Springer, 1035--1069.

[38]

I.M. Sobol. 1967. On the distribution of points in a cube and the approximate evaluation of integrals. U. S. S. R. Comput. Math. and Math. Phys. 7, 4 (Jan. 1967), 86--112.

[39]

J.C. Spall. 1992. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans. Automat. Control 37, 3 (1992), 332--341.

[40]

R. Storn and K. Price. 1997. Differential Evolution - A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces. Journal of Global Optimization 11, 4 (1997), 341--359.

Digital Library

[41]

H. Wang, D. Vermetten, F. Ye, C. Doerr, and T. Bäck. 2022. IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics. ACM Trans. Evol. Learn. Optim. 2, Article 3 (2022).

Digital Library

[42]

P. Winker and K.T. Fang. 1997. Applications of Threshold-Accepting to the evaluation of the discrepancy of a set of points. SIAM J. Numerical Analysis 34 (1997), 2028--2042.

Digital Library

Cited By

Rusch TKirk NBronstein MLemieux CRus D(2024)Message-Passing Monte Carlo: Generating low-discrepancy point sets via graph neural networksProceedings of the National Academy of Sciences10.1073/pnas.2409913121121:40Online publication date: 26-Sep-2024
https://doi.org/10.1073/pnas.2409913121
Vermetten DLengler JRusin DBäck TDoerr C(2024)Empirical Analysis of the Dynamic Binary Value Problem with IOHprofilerParallel Problem Solving from Nature – PPSN XVIII10.1007/978-3-031-70068-2_2(20-35)Online publication date: 7-Sep-2024
https://doi.org/10.1007/978-3-031-70068-2_2

Index Terms

Computing Star Discrepancies with Numerical Black-Box Optimization Algorithms
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies
      1. Randomized search
  2. Parallel computing methodologies
    1. Parallel algorithms

Recommendations

AMaLGaM IDEAs in noisy black-box optimization benchmarking
GECCO '09: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers

This paper describes the application of a Gaussian Estimation-of-Distribution (EDA) for real-valued optimization to the noisy part of a benchmark introduced in 2009 called BBOB (Black-Box Optimization Benchmarking). Specifically, the EDA considered here ...
AMaLGaM IDEAs in noiseless black-box optimization benchmarking
GECCO '09: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers

This paper describes the application of a Gaussian Estimation-of-Distribution (EDA) for real-valued optimization to the noiseless part of a benchmark introduced in 2009 called BBOB (Black-Box Optimization Benchmarking). Specifically, the EDA considered ...
Black-box optimization benchmarking for noiseless function testbed using particle swarm optimization
GECCO '09: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers

This paper benchmarks the Particle Swarm Optimization (PSO) algorithm using the noise-free BBOB 2009 testbed.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '23: Proceedings of the Genetic and Evolutionary Computation Conference

July 2023

1667 pages

ISBN:9798400701191

DOI:10.1145/3583131

Chair:
Sara Silva,
Program Chair:
Luís Paquete

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

GECCO '23

Sponsor:

SIGEVO

GECCO '23: Genetic and Evolutionary Computation Conference

July 15 - 19, 2023

Lisbon, Portugal

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
52
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)3

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rusch TKirk NBronstein MLemieux CRus D(2024)Message-Passing Monte Carlo: Generating low-discrepancy point sets via graph neural networksProceedings of the National Academy of Sciences10.1073/pnas.2409913121121:40Online publication date: 26-Sep-2024
https://doi.org/10.1073/pnas.2409913121
Vermetten DLengler JRusin DBäck TDoerr C(2024)Empirical Analysis of the Dynamic Binary Value Problem with IOHprofilerParallel Problem Solving from Nature – PPSN XVIII10.1007/978-3-031-70068-2_2(20-35)Online publication date: 7-Sep-2024
https://doi.org/10.1007/978-3-031-70068-2_2

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents