Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
article
Free access

Evolutionary Model Type Selection for Global Surrogate Modeling

Published: 01 December 2009 Publication History

Abstract

Due to the scale and computational complexity of currently used simulation codes, global surrogate (metamodels) models have become indispensable tools for exploring and understanding the design space. Due to their compact formulation they are cheap to evaluate and thus readily facilitate visualization, design space exploration, rapid prototyping, and sensitivity analysis. They can also be used as accurate building blocks in design packages or larger simulation environments. Consequently, there is great interest in techniques that facilitate the construction of such approximation models while minimizing the computational cost and maximizing model accuracy. Many surrogate model types exist (Support Vector Machines, Kriging, Neural Networks, etc.) but no type is optimal in all circumstances. Nor is there any hard theory available that can help make this choice. In this paper we present an automatic approach to the model type selection problem. We describe an adaptive global surrogate modeling environment with adaptive sampling, driven by speciated evolution. Different model types are evolved cooperatively using a Genetic Algorithm (heterogeneous evolution) and compete to approximate the iteratively selected data. In this way the optimal model type and complexity for a given data set or simulation code can be dynamically determined. Its utility and performance is demonstrated on a number of problems where it outperforms traditional sequential execution of each model type.

References

[1]
E. Alba and M. Tomassini. Parallelism and evolutionary algorithms. IEEE Trans. Evolutionary Computation, 6(5):443-462, 2002.
[2]
C. A. Ankenbrandt. An extension to the theory of convergence and a proof of the time complexity of genetic algorithms. In Foundations of Genetic Algorithms, pages 53-68, 1990.
[3]
Y. B., R. El-Yaniv, and K. Luz. Online choice of active learning algorithms. Journal of Machine Learning Research, 5:255-291, 2004.
[4]
D. Busby, C. L. Farmer, and A. Iske. Hierarchical nonlinear approximation for experimental design and statistical data fitting. SIAM Journal on Scientific Computing, 29(1):49-69, 2007.
[5]
P-W. Chen, J-Y. Wang, and H-M. Lee. Model selection of SVMs using GA approach. In Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on, volume 3, pages 2035-2040, 25-29 July 2004.
[6]
V. Chen, K-L. Tsui, R. Barton, and M. Meckesheimer. A review on design, modeling and applications of computer experiments. IIE Transactions, 38:273-291, 2006.
[7]
H-S. Chung and J. J. Alonso. Comparison of approximation models with merit functions for design optimization. In 8th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Long Beach, CA, September 2000. AIAA Paper 2000-4754.
[8]
S. Conti and A. O'Hagan. Bayesian emulation of complex multi-output and dynamic computer models. research report no. 569/07, submitted to Journal of Statistical Planning and Inference. Technical report, Department of Probability and Statistics, University of Sheffield, 2007.
[9]
I. Couckuyt, D. Gorissen, H. Rouhani, E. Laermans, and T. Dhaene. Evolutionary regression modeling with active learning: An application to rainfall runoff modeling. In International Conference on Adaptive and Natural Computing Algorithms, volume LNCS 5495, pages 548-558, 2009.
[10]
K. Crombecq, D. Gorissen, L. De Tommasi, and T. Dhaene. A novel sequential design strategy for global surrogate modeling. In Proceedings of the 41th Conference on Winter Simulation, accepted, 2009.
[11]
D. Deschrijver and T. Dhaene. Rational modeling of spectral data using orthonormal vector fitting. In Signal Propagation on Interconnects, 2005. Proceedings. 9th IEEE Workshop on, pages 111- 114, 10-13 May 2005.
[12]
M. S. Eldred and D. M. Dunlavy. Formulations for surrogate-based optimization wiht data fit, multifidelity, and reduced-order models. In 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Protsmouth, Virginia, 2006.
[13]
M. T. M. Emmerich, K. Giannakoglou, and B. Naujoks. Single- and multiobjective evolutionary optimization assisted by gaussian random field metamodels. IEEE Trans. Evolutionary Computation, 10(4):421-439, 2006.
[14]
H. J. Escalante, M. M. Gomez, and L. E. Sucar. PSMS for neural networks. In The IJCNN 2007 Agnostic vs Prior Knowledge Challenge, pages 678-683, 2007.
[15]
H. Jair Escalante, M. Montes, and E. Sucar. Particle swarm model selection. Journal of Machine Learning Research (special issue on model selection), 10:405-440, 2008.
[16]
H. Fang, M. Rais-Rohani, Z. Liu, and M.F. Horstemeyer. A comparative study of metamodeling methods for multiobjective crashworthiness optimization. Computers and Structures, 83:2121- 2136, 2005.
[17]
F.D. Foresee andM.T. Hagan. Gauss-newton approximation to bayesian regularization. In Proceedings of the 1997 International Joint Conference on Neural Networks, pages 1930-1935, 1997.
[18]
A. Forrester, A. Sobester, and A. Keane. Engineering Design Via Surrogate Modelling: A Practical Guide. Wiley, 2008.
[19]
F. Friedrichs and C. Igel. Evolutionary tuning of multiple svm parameters. Neurocomputing, 64: 107-117, 2005.
[20]
S.E. Gano, H. Kim, and D.E. Brown. Comparison of three surrogate modeling techniques: Datascape, kriging, and second order regression. In Proceedings of the 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, AIAA-2006-7048, Portsmouth, Virginia, 2006.
[21]
M. Ganser, K. Grossenbacher, M. Schutz, L. Willmes, and T. Back. Simulation meta-models in the early phases of the product development process. In Proceedings of Efficient Methods for Robust Design and Optimization (EUROMECH 07), 2007.
[22]
K. C. Giannakoglou, M. K. Karakasis, and I. C. Kampolis. Evolutionary algorithms with surrogate modeling for computationally expensive optimization problem. In Proceedings of ERCOFTAC 2006 Design Optimization International Conference, Gran Canaria, Spain, 2006.
[23]
T. Goel, R. Haftka, W. Shyy, and N. Queipo. Ensemble of surrogates. Structural and Multidisciplinary Optimization, 33:199-216, 2007.
[24]
D. Goldberg. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Professional, January 1989. ISBN 0201157675.
[25]
D. Gorissen. Heterogeneous evolution of surrogate models. Master's thesis, Master of AI, Katholieke Universiteit Leuven (KUL), 2007.
[26]
D. Gorissen, L. De Tommasi, J. Croon, and T. Dhaene. Automatic model type selection with heterogeneous evolution: An application to RF circuit block modeling. In Proceedings of the IEEE Congress on Evolutionary Computation, WCCI 2008, Hong Kong, 2008a.
[27]
D. Gorissen, L. De Tommasi, W. Hendrickx, J. Croon, and T. Dhaene. RF circuit block modeling via kriging surrogates. In Proceedings of the 17th International Conference on Microwaves, Radar and Wireless Communications (MIKON 2008), 2008b.
[28]
D. Gorissen, I. Couckuyt, K. Crombecq, and T. Dhaene. Pareto-based multi-output model type selection. In Proceedings of the 4th International Conference on Hybrid Artificial Intelligence (HAIS 2009), Salamanca, Spain, pages 442-449. Springer - Lecture Notes in Artificial Intelligence, Vol. LNCS 5572, 2009a.
[29]
D. Gorissen, I. Couckuyt, E. Laermans, and T. Dhaene. Multiobjective global surrogate modeling, dealing with the 5-percent problem - in press. Engineering with Computers, 2009b.
[30]
D. Gorissen, L. De Tommasi, K. Crombecq, and T. Dhaene. Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Computing and Applications, 18(5): 485-494, 2009c.
[31]
R. Gramacy, H. Lee, and W. Macready. Parameter space exploration with gaussian process trees. In ICML '04: Proceedings of the 21st International Conference on Machine Learning, page 45, New York, NY, USA, 2004. ACM Press. ISBN 1-58113-828-5. 1015330.1015367.
[32]
L. Gu. A comparison of polynomial based regression models in vehicle safety analysis. In A. Diaz, editor, 2001 ASME Design Automation Conference, ASME, Pittsburgh, PA, 2001.
[33]
D. Harrison and D.L. Rubinfeld. Hedonic prices and the demand for clean air. Journal of Environmental Economics & Management, 5:81-102, 1978.
[34]
W. Hendrickx, D. Gorissen, and T. Dhaene. Grid enabled sequential design and adaptive metamodeling. In WSC '06: Proceedings of the 37th Conference on Winter simulation, pages 872-881. Winter Simulation Conference, 2006. ISBN 1-4244-0501-7.
[35]
C. Hocaoglu and A. Sanderson. Planning multiple paths with evolutionary speciation. IEEE Trans. Evolutionary Computation, 5(3):169-191, 2001.
[36]
J. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, 1975.
[37]
R. Jin, W. Chen, and T.W. Simpson. Comparative studies of metamodelling techniques under multiple modelling criteria. Structural and Multidisciplinary Optimization, 23(1):1-13, December 2001.
[38]
Yaochu Jin and B. Sendhoff. Pareto-based multiobjective machine learning: An overview and case studies. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 38(3):397-415, May 2008. ISSN 1094-6977.
[39]
Yaochu Jin, Markus Olhofer, and Bernhard Sendhoff. A framework for evolutionary optimization with approximate fitness functions. IEEE Trans. Evolutionary Computation, 6(5):481-494, 2002.
[40]
D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive blackbox functions. J. of Global Optimization, 13(4):455-492, 1998. ISSN 0925-5001. //dx.doi.org/10.1023/A:1008306431147.
[41]
A. C. Keys, L. P. Rees, and A. G. Greenwood. Performance measures for selection of metamodels to be used in simulation optimization. Decision Sciences, 33:31-58, 2007.
[42]
J. Kleijnen, S. Sanchez, T. Lucas, and T. Cioppa. State-of-the-art review: A user's guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing, 17(3): 263-289, 2005.
[43]
J. Knowles and H. Nakayama. Meta-modeling in multiobjective optimization. In Multiobjective Optimization: Interactive and Evolutionary Approaches, pages 245-284. Springer-Verlag, Berlin, Heidelberg, 2008. ISBN 978-3-540-88907-6. 978-3-540-88908-3_10.
[44]
R. Lehmensiek. Efficient Adaptive Sampling Applied to Multivariate, Multiple Output Rational InterpolationModels, with Applications in Electromagnetics-based Device Modeling. PhD thesis, University of Stellenbosch, 2001.
[45]
S. Lessmann, R. Stahlbock, and S.F. Crone. Genetic algorithms for support vector machine model selection. In Proceedings of the International Joint Conference on Neural Networks, 2006. IJCNN '06., pages 3063-3069, 16-21 July 2006.
[46]
X. Rong Li and Z. Zhao. Evaluation of estimation algorithms part I: incomprehensive measures of performance. IEEE Transactions on Aerospace and Electronic Systems, 42(4):1340-1358, October 2006. ISSN 0018-9251.
[47]
D. Lim, Y-S. Ong, Y. Jin, and B. Sendhoff. A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation. In Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO 07), pages 1288-1295, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-697-4.
[48]
S. N. Lophaven, H. B. Nielsen, and J. Søndergaard. Aspects of the matlab toolbox DACE. Technical report, Informatics and Mathematical Modelling, Technical University of Denmark, DTU, Richard Petersens Plads, Building 321, DK-2800 Kgs. Lyngby, 2002.
[49]
D. MacKay. Bayesian model comparison and backprop nets. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 839-846, San Mateo, California, 1992. Morgan Kaufmann.
[50]
T. Nakama. Theoretical analysis of genetic algorithms in noisy environments based on a markov model. In Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation (GECCO 08), pages 1001-1008, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-130-9.
[51]
A. Neubauer. A theoretical analysis of the non-uniform mutation operator for the modified genetic algorithm. In Proceedings of the IEEE International Conference on Evolutionary Computation, pages 93-96, Apr 1997.
[52]
M. Nowostawski and R. Poli. Parallel genetic algorithm taxonomy. In Knowledge-Based Intelligent Information Engineering Systems, 1999. Third International Conference, pages 88-92, 31 Aug.-1 Sept. 1999.
[53]
Y-S. Ong, P.B. Nair, and K.Y. Lum. Max-min surrogate-assisted evolutionary algorithm for robust design. Evolutionary Computation, IEEE Transactions on, 10(4):392-404, Aug. 2006.
[54]
I. Paenke, J. Branke, and Y. Jin. Efficient search for robust solutions by means of evolutionary algorithms and fitness approximation. IEEE Trans. Evolutionary Computation, 10(4):405-420, 2006.
[55]
B. Pamadi, P. Covell, P. Tartabini, and K. Murphy. Aerodynamic characteristics and glide-back performance of langley glide-back booster. In Proceedings of 22nd Applied Aerodynamics Conference and Exhibit, Providence, Rhode Island, 2004.
[56]
H. Pei and E. Goodman. A comparison of cohort genetic algorithms with canonical serial and island-model distributed ga's. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), pages 501-510, San Francisco, California, USA, 7-11 July 2001. Morgan Kaufmann. ISBN 1-55860-774-9.
[57]
R. Poli. Exact schema theory for genetic programming and variable-length genetic algorithms with one-point crossover. Genetic Programming and Evolvable Machines, 2(2):123-163, 2001. ISSN 1389-2576.
[58]
X. Qi and F. Palmieri. Theoretical analysis of evolutionary algorithms with an infinite population size in continuous space. part I: Basic properties of selection and mutation. Neural Networks, IEEE Transactions on, 5(1):102-119, Jan 1994a. ISSN 1045-9227.
[59]
X. Qi and F. Palmieri. Theoretical analysis of evolutionary algorithms with an infinite population size in continuous space. part II: Analysis of the diversification role of crossover. Neural Networks, IEEE Transactions on, 5(1):120-129, Jan 1994b. ISSN 1045-9227.
[60]
N. Queipo, R. Haftka, W. Shyy, T. Goel, R. Vaidyanathan, and P. Tucker. Surrogate-based analysis and optimization. Progress in Aerospace Sciences, 41:1-28, 2005.
[61]
C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[62]
G. Rawlins, editor. Foundations of Genetic Algorithms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1991. ISBN 1558605592.
[63]
R.G. Regis and C.A. Shoemaker. Local function approximation in evolutionary algorithms for the optimization of costly functions. IEEE Trans. Evolutionary Computation, 8(5):490-505, 2004.
[64]
S. Rogers, M. Aftosmis, S. Pandya, and N. Chaderjian. Automated CFD parameter studies on distributed parallel computers. In Proc of 16th AIAA Computational Fluid Dynamics Conference, Orlando, Florida, 2003.
[65]
E. Sanchez, S. Pintos, and N.V. Queipo. Toward an optimal ensemble of kernel-based approximations with engineering applications. In In Proceedings of the International Joint Conference on Neural Networks, 2006. IJCNN '06., pages 2152-2158, 2006.
[66]
T. Santner, B. Williams, and W. Notz. The design and analysis of computer experiments. Springer series in statistics. Springer, 2003.
[67]
A. Sharkey. On combining artificial neural nets. Connectionist Science, 8(3):299-314, 1996.
[68]
T. Simpson, J. D. Poplinski, P. N. Koch, and J. K. Allen. Metamodels for computer-based engineering design: Survey and recommendations. Eng. Comput. (Lond.), 17(2):129-150, 2001.
[69]
T. W. Simpson, V. Toropov, V. Balabanov, and F. A. C. Viana. Design and analysis of computer experiments in multidisciplinary design optimization: a review of how far we have come or not. In Proceedings of the 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2008 MAO, Victoria, Canada, 2008.
[70]
G. Smits and M. Kotanchek. Pareto-front exploitation in symbolic regression. In Genetic Programming Theory and Practice II. Springer, Ann Arbor, USA, 2004.
[71]
D. P. Solomatine and A. Ostfeld. Data-driven modelling : some past experiences and new approaches. Journal of hydroinformatics, 10(1):3-22, 2008.
[72]
K. Stanley and R. Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2):99-127, 2002. ISSN 1063-6560. 106365602320169811.
[73]
M. Streeter and L. Becker. Automated discovery of numerical approximation formulae via genetic programming. Genetic Programming and Evolvable Machines, 4(3):255-286, 2003. ISSN 1389- 2576.
[74]
J.A.K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, and J. Vandewalle. Least Squares Support Vector Machines. World Scientific Publishing Co., Pte, Ltd., Singapore, 2002. ISBN 981-238-151-1.
[75]
D. J.J. Toal, N. W. Bressloff, and A. J. Keane. Kriging hyperparameter tuning strategies. AIAA Journal, 46(5):1240-1252, 2008.
[76]
S. Tomioka, S. Nisiyama, and T. Enoto. Nonlinear least square regression by adaptive domain method with multiple genetic algorithms. IEEE Transactions on Evolutionary Computation, 11 (1):1-16, February 2007.
[77]
P. Triverio, S. Grivet-Talocia, M.S. Nakhla, F. G. Canavero, and R. Achar. Stability, causality, and passivity in electrical interconnect models. IEEE Transactions on Advanced Packaging, 30(4): 795-808, 2007.
[78]
C. Turner, R. Crawford, and M. Campbell. Multidimensional sequential sampling for nurbs-based metamodel development. Engineering with Computers, 23(3):155-174, 2007. ISSN 0177-0667.
[79]
E.J. Vladislavleva, G.F. Smits, and D. den Hertog. Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming. Evolutionary Computation, IEEE Transactions on, 13(2):333-349, April 2009. ISSN 1089-778X. 1109/TEVC.2008.926486.
[80]
I. Voutchkov and A.J. Keane. Multiobjective Optimization using Surrogates. In I.C. Parmee, editor, Adaptive Computing in Design and Manufacture 2006. Proceedings of the Seventh International Conference, pages 167-175, Bristol, UK, April 2006.
[81]
G. Wang and S. Shan. Review of metamodeling techniques in support of engineering design optimization. Journal of Mechanical Design, 129(4):370-380, 2007.
[82]
D. Whitley, S. Rana, and R. B. Heckendorn. The island model genetic algorithm: On separability, population size and convergence. Journal of Computing and Information Technology, 7(1):33-47, 1999.
[83]
Y. Xiong, W . Chen, D. Apley, and X. Ding. A nonstationary covariance-based kriging method for metamodeling in engineering design. International Journal for Numerical Methods in Engineering, 71(6):733-756, 2007.
[84]
R.J. Yang, N. Wang, C. H. Tho, and J. P. Bobinaeu. Metamodeling development for vehicle frontal impact simulation. Journal of Mechanical Design, 127(5):1014-1020, September 2005.
[85]
X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423-1447, Sept. 1999.
[86]
X. Yao and Y. Xu. Recent advances in evolutionary computation. Journal of Computer Science and Technology, 21(1):1-18, 2006.
[87]
K. Ye, W. Li, and A. Sudjianto. Algorithmic construction of optimal symmetric latin hypercube designs. Journal of Statistical Planning and Inference, 90:145-159, 2000.
[88]
Y-S. Yeun, W-S. Ruy, Y-S. Yang, and N-J. Kim. Implementing linear models in genetic programming. IEEE Trans. Evolutionary Computation, 8(6):542-566, 2004.
[89]
C. Zhang, H. Shao, and Y. Li. Particle swarm optimisation for evolving artificial neural network. In Systems, Man, and Cybernetics, 2000 IEEE International Conference on, volume 4, pages 2487-2490vol.4, 8-11 Oct. 2000.

Cited By

View all
  • (2020)Reduced-Cost Constrained Modeling of Microwave and Antenna Components: Recent AdvancesComputational Science – ICCS 202010.1007/978-3-030-50426-7_4(40-56)Online publication date: 3-Jun-2020
  • (2019)Solving challenges at the interface of simulation and big data using machine learningProceedings of the Winter Simulation Conference10.5555/3400397.3400444(572-583)Online publication date: 8-Dec-2019
  • (2018)Automatic selection for general surrogate modelsStructural and Multidisciplinary Optimization10.1007/s00158-018-1925-358:2(719-734)Online publication date: 1-Aug-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research
The Journal of Machine Learning Research  Volume 10, Issue
12/1/2009
2936 pages
ISSN:1532-4435
EISSN:1533-7928
Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 December 2009
Published in JMLR Volume 10

Qualifiers

  • Article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)24
  • Downloads (Last 6 weeks)6
Reflects downloads up to 24 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2020)Reduced-Cost Constrained Modeling of Microwave and Antenna Components: Recent AdvancesComputational Science – ICCS 202010.1007/978-3-030-50426-7_4(40-56)Online publication date: 3-Jun-2020
  • (2019)Solving challenges at the interface of simulation and big data using machine learningProceedings of the Winter Simulation Conference10.5555/3400397.3400444(572-583)Online publication date: 8-Dec-2019
  • (2018)Automatic selection for general surrogate modelsStructural and Multidisciplinary Optimization10.1007/s00158-018-1925-358:2(719-734)Online publication date: 1-Aug-2018
  • (2018)Concurrent surrogate model selection (COSMOS)Structural and Multidisciplinary Optimization10.1007/s00158-017-1797-y57:3(1093-1114)Online publication date: 1-Mar-2018
  • (2017)New coral reefs-based approaches for the model type selection problemInternational Journal of Bio-Inspired Computation10.1504/IJBIC.2017.08669810:3(145-158)Online publication date: 1-Jan-2017
  • (2017)Fog Density Estimation and Image Defogging Based on Surrogate Modeling for Optical DepthIEEE Transactions on Image Processing10.1109/TIP.2017.270072026:7(3397-3409)Online publication date: 1-Jul-2017
  • (2016)Development of a surrogate-based vehicle dynamic model to reduce computational delays in a driving simulatorSimulation10.1177/003754971667595692:12(1087-1102)Online publication date: 1-Dec-2016
  • (2016)Performance comparison of several response surface surrogate models and ensemble methods for water injection optimization under uncertaintyComputers & Geosciences10.1016/j.cageo.2016.02.02291:C(19-32)Online publication date: 1-Jun-2016
  • (2015)Surrogate-assisted multi-objective model selection for support vector machinesNeurocomputing10.1016/j.neucom.2014.08.075150:PA(163-172)Online publication date: 20-Feb-2015
  • (2015)Initial sampling methods in metamodel-assisted optimizationEngineering with Computers10.1007/s00366-014-0372-z31:4(661-680)Online publication date: 1-Oct-2015
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media