Transfer Learning of Surrogate Models via Domain Affine Transformation
Authors: 
Shuaiqun Pan, Diederick Vermetten, 
Manuel López-Ibáñez, 
Thomas Bäck, Hao WangAuthors Info & Claims
Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao WangAuthors Info & ClaimsPages 385 - 393
Abstract
Surrogate models are widely applied in many scenarios to replace expensive executions of real-world procedures. Training a high-quality surrogate model often requires many sample points, which can be costly to obtain. We would amortize this cost if we could reuse already-trained surrogates in future tasks, provided certain invariances are retained across tasks. This paper studies transferring a surrogate model trained on a source function to a target function using a small data set. As a first step, we consider the following invariance: the domains of the source and target functions are related by an unknown affine transformation. We propose to parameterize the surrogate of the source with an affine transformation and optimize it w.r.t. an empirical loss measured with a small transfer data set sampled on the target. We select all functions from the well-known black-box optimization benchmark (BBOB) as the source and artificially generate the target with affine transformation sampled u.a.r. We experiment with a commonly used surrogate model, Gaussian process regression, where results show that the transferred surrogate significantly outperforms both the original surrogate and the one built from scratch with the transfer data set.
References
[1]
Thomas Bäck. 1996. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford University Press.
[2]
Thomas Bäck, Christophe Foussette, and Peter Krause. 2013. Contemporary Evolution Strategies. Springer.
[3]
Atharv Bhosekar and Marianthi Ierapetritou. 2018. Advances in surrogate based modeling, feasibility analysis, and optimization: A review. Comput. Chem. Eng. 108 (2018), 250--267.
[4]
Adam D. Bull. 2011. Convergence Rates of Efficient Global Optimization Algorithms. J. Mach. Learn. Res. 12 (2011), 2879--2904.
[5]
Bin Cao, Sinno Jialin Pan, Yu Zhang, Dit-Yan Yeung, and Qiang Yang. 2010. Adaptive Transfer Learning. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010, Maria Fox and David Poole (Eds.). AAAI Press. http://www.aaai.org/ocs/index.php/AAAI/AAAI10/paper/view/1823
[6]
Gengxiang Chen, Yingguang Li, and Xu Liu. 2019. Pose-dependent tool tip dynamics prediction using transfer learning. International Journal of Machine Tools and Manufacture 137 (2019), 30--41.
[7]
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc'Aurelio Ranzato, Sagi Perel, and Nando de Freitas. 2022. Towards Learning Universal Hyperparameter Optimizers with Transformers. In Advances in Neural Information Processing Systems. http://papers.nips.cc/paper_files/paper/2022/hash/cf6501108fced72ee5c47e2151c4e153-Abstract-Conference.html
[8]
Kai Cheng and Zhenzhou Lu. 2021. Adaptive Bayesian support vector regression model for structural reliability analysis. Reliab. Eng. Syst. Saf. 206 (2021), 107286.
[9]
Hal Daumé. 2009. Frustratingly Easy Domain Adaptation. CoRR abs/0907.1815 (2009). arXiv:0907.1815 http://arxiv.org/abs/0907.1815
[10]
Jacob de Nobel, Furong Ye, Diederick Vermetten, Hao Wang, Carola Doerr, and Thomas Bäck. 2024. IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics. Evolutionary Computation (2024), 1--6.
[11]
Michael Emmerich, Ofer M. Shir, and Hao Wang. 2018. Evolution Strategies. In Handbook of Heuristics, Rafael Martí, Panos M. Pardalos, and Mauricio G. C. Resende (Eds.). Springer, 89--119.
[12]
Jianguang Fang, Guangyong Sun, Na Qiu, Nam H Kim, and Qing Li. 2017. On design optimization for structural crashworthiness and its state of the art. Structural and Multidisciplinary Optimization 55 (2017), 1091--1119.
[13]
Matthias Feurer and Frank Hutter. 2019. Hyperparameter Optimization. In Automated Machine Learning, Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (Eds.). Springer, 3--33.
[14]
Benito E. Flores. 1986. A pragmatic view of accuracy measurement in forecasting. Omega 14, 2 (1986), 93--98.
[15]
Alexander I. J. Forrester and Andy J. Keane. 2009. Recent advances in surrogate-based optimization. Progress in Aerospace Sciences 45, 1-3 (2009), 50--79.
[16]
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. 2020. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software 36, 1 (2020), 1--31.
[17]
Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Technical Report RR-6829. INRIA, France. Updated February 2010.
[18]
Marika Kaden, Mandy Lange, David Nebel, Martin Riedel, Tina Geweniger, and Thomas Villmann. 2014. Aspects in classification learning - Review of recent developments in Learning Vector Quantization. Foundations of Computing and Decision Sciences 39, 2 (2014), 79--105.
[19]
Pascal Kerschke and Heike Trautmann. 2019. Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation 27, 1 (2019), 99--127.
[20]
Morteza Kiani and Ali R Yildiz. 2016. A comparative study of non-traditional methods for vehicle crashworthiness and NVH optimization. Archives of Computational Methods in Engineering 23 (2016), 723--734.
[21]
Aaron Klein, Zhenwen Dai, Frank Hutter, Neil D. Lawrence, and Javier González. 2019. Meta-Surrogate Benchmarking for Hyperparameter Optimization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 6267--6277. https://proceedings.neurips.cc/paper/2019/hash/0668e20b3c9e9185b04b3d2a9dc8fa2d-Abstract.html
[22]
Teuvo Kohonen. 1990. Improved versions of learning vector quantization. In Proceedings of the International Joint Conference on Neural Networks (IJCNN). IEEE, 545--550.
[23]
Marius Thomas Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. 2022. SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. Journal of Machine Learning Research 23 (2022), 1--9.
[24]
Fu Xing Long, Bas van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas Bäck. 2022. Learning the characteristics of engineering optimization problems with applications in automotive crash. In GECCO '22: Genetic and Evolutionary Computation Conference, Boston, Massachusetts, USA, July 9 - 13, 2022, Jonathan E. Fieldsend and Markus Wagner (Eds.). ACM, 1227--1236.
[25]
Luis Mandl, André Mielke, Seyed Morteza Seyedpour, and Tim Ricken. 2023. Affine transformations accelerate the training of physics-informed neural networks of a one-dimensional consolidation problem. Scientific Reports 13, 1 (2023), 15566.
[26]
Shunya Minami, Kenji Fukumizu, Yoshihiro Hayashi, and Ryo Yoshida. 2023. Transfer Learning with Affine Model Transformation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/3819a070922cc0d19f3d66ce108f28e0-Abstract-Conference.html
[27]
Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, and Hao Wang. 2024. Transfer Learning of Surrogate Models via Domain Affine Transformation: Supplementary Material.
[28]
Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. 2009. Domain Adaptation via Transfer Component Analysis. In IJCAI 2009, Proceedings of the 21st International Joint Conference on Artificial Intelligence, Pasadena, California, USA, July 11-17, 2009, Craig Boutilier (Ed.). 1187--1192. http://ijcai.org/Proceedings/09/Papers/200.pdf
[29]
Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. 2011. Domain Adaptation via Transfer Component Analysis. IEEE Trans. Neural Networks 22, 2 (2011), 199--210.
[30]
Milan Papez and Anthony Quinn. 2022. Transferring model structure in Bayesian transfer learning for Gaussian process regression. Knowl. Based Syst. 251 (2022), 108875.
[31]
David Pardoe and Peter Stone. 2010. Boosting for regression transfer. In Proceedings of the 27th International Conference on International Conference on Machine Learning. 863--870.
[32]
Julien Pelamatti, Loïc Brevault, Mathieu Balesdent, El-Ghazali Talbi, and Yannick Guerin. 2020. Overview and Comparison of Gaussian Process-Based Surrogate Models for Mixed Continuous and Discrete Variables: Application on Aerospace Design Problems. High-Performance Simulation-Based Optimization (2020), 189--224.
[33]
Na Qiu, Yunkai Gao, Jianguang Fang, Guangyong Sun, Qing Li, and Nam H Kim. 2018. Crashworthiness optimization with uncertainty from surrogate model and numerical error. Thin-Walled Structures 129 (2018), 457--472.
[34]
Dushhyanth Rajaram, Tejas G Puranik, S Ashwin Renganathan, WoongJe Sung, Olivia Pinon Fischer, Dimitri N Mavris, and Arun Ramamurthy. 2021. Empirical Assessment of Deep Gaussian Process Surrogate Models for Engineering Problems. Journal of Aircraft 58, 1 (2021), 182--196.
[35]
Elena Raponi, Nathanaël Carraz Rakotonirina, Jérémy Rapin, Carola Doerr, and Olivier Teytaud. 2023. Optimizing with low budgets: A comparison on the black-box optimization benchmarking suite and openai gym. IEEE Transactions on Evolutionary Computation (2023).
[36]
Elena Raponi, Hao Wang, Mariusz Bujny, Simonetta Boria, and Carola Doerr. 2020. High Dimensional Bayesian Optimization Assisted by Principal Component Analysis. In Parallel Problem Solving from Nature - PPSN XVI, Thomas Bäck, Mike Preuss, André Deutz, Hao Wang, Carola Doerr, Michael T. M. Emmerich, and Heike Trautmann (Eds.). Lecture Notes in Computer Science, Vol. 12269. Springer, Cham, Switzerland, 169--183.
[37]
Taisei Saida and Mayuko Nishio. 2023. Transfer learning Gaussian process regression surrogate model with explainability for structural reliability analysis under variation in uncertainties. Computers & Structures 281 (2023), 107014.
[38]
Sascha Saralajew and Thomas Villmann. 2017. Transfer learning in classification based on manifolc. models and its relation to tangent metric learning. In 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, 1756--1765.
[39]
Priyanka Singh and Pragya Dwivedi. 2018. Integration of new evolutionary approach with artificial neural network for solving short term load forecast problem. Applied energy 217 (2018), 537--549.
[40]
Urban Skvorc, Tome Eftimov, and Peter Korosec. 2020. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Applied Soft Computing 90 (2020), 106138.
[41]
Jasper Snoek, Kevin Swersky, Richard Zemel, and Ryan P. Adams. 2014. Input Warping for Bayesian Optimization of Non-Stationary Functions. In Proceedings of the 31st International Conference on Machine Learning, ICML 2014, Eric P. Xing and Tony Jebara (Eds.), Vol. 32. PMLR, 1674--1682. http://jmlr.org/proceedings/papers/v32/
[42]
Baochen Sun, Jiashi Feng, and Kate Saenko. 2017. Correlation Alignment for Unsupervised Domain Adaptation. In Domain Adaptation in Computer Vision Applications, Gabriela Csurka (Ed.). Springer, 153--171.
[43]
Ye Tian, Shichen Peng, Xingyi Zhang, Tobias Rodemann, Kay Chen Tan, and Yaochu Jin. 2020. A Recommender System for Metaheuristic Algorithms for Continuous Optimization Based on Deep Recurrent Neural Networks. IEEE Transactions on Artificial Intelligence 1, 1 (2020), 5--18.
[44]
Rohit Tripathy and Ilias Bilionis. 2018. Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification. Journal of computational physics 375 (2018), 565--588.
[45]
Aad W van der Vaart and J Harry van Zanten. 2008. Rates of contraction of posterior distributions based on Gaussian process priors. (2008).
[46]
Ky Khac Vu, Claudia D'Ambrosio, Youssef Hamadi, and Leo Liberti. 2017. Surrogate-based methods for black-box optimization. International Transactions in Operational Research 24, 3 (2017), 393--424.
[47]
Xilu Wang, Yaochu Jin, Sebastian Schmitt, Markus Olhofer, and Richard All-mendinger. 2021. Transfer learning based surrogate assisted evolutionary biobjective optimization for objectives with different evaluation times. Knowledge-Based Systems 227 (2021), 107190.
[48]
Kaifeng Yang and Michael Affenzeller. 2023. Surrogate-assisted Multi-objective Optimization via Genetic Programming Based Symbolic Regression. In Evolutionary Multi-Criterion Optimization - 12th International Conference, EMO 2023, Leiden, The Netherlands, March 20-24, 2023, Proceedings (Lecture Notes in Computer Science, Vol. 13970), Michael Emmerich, André H. Deutz, Hao Wang, Anna V. Kononova, Boris Naujoks, Ke Li, Kaisa Miettinen, and Iryna Yevseyeva (Eds.). Springer, 176--190.
[49]
Kai Yu, Volker Tresp, and Anton Schwaighofer. 2005. Learning Gaussian processes from multiple tasks. In Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005 (ACM International Conference Proceeding Series, Vol. 119), Luc De Raedt and Stefan Wrobel (Eds.). ACM, 1012--1019.
[50]
Paolo Zanini, Marco Congedo, Christian Jutten, Salem Said, and Yannick Berthoumieu. 2017. Transfer learning: A Riemannian geometry framework with applications to brain-computer interfaces. IEEE Transactions on Biomedical Engineering 65, 5 (2017), 1107--1116.
[51]
Xi Zhang, Guo Yu, Yaochu Jin, and Feng Qian. 2023. An adaptive Gaussian process based manifold transfer learning to expensive dynamic multi-objective optimization. Neurocomputing 538 (2023), 126212.
Index Terms
- Transfer Learning of Surrogate Models via Domain Affine Transformation
Recommendations
Transfer active learning
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge managementActive learning traditionally assumes that labeled and unlabeled samples are subject to the same distributions and the goal of an active learner is to label the most informative unlabeled samples. In reality, situations may exist that we may not have ...
Transfer prior knowledge from surrogate modelling: A meta-learning approach
Highlights- Propose a meta-learning-based surrogate modelling (MLSM) framework for knowledge transfer.
AbstractSurrogate modelling has emerged as a useful technique to study complex physical and engineering systems in various disciplines, especially for engineering analysis. Previous studies mostly focused on developing new surrogate models and/...
Transfer learning with affine model transformation
NIPS '23: Proceedings of the 37th International Conference on Neural Information Processing SystemsSupervised transfer learning has received considerable attention due to its potential to boost the predictive power of machine learning in scenarios where data are scarce. Generally, a given set of source models and a dataset from a target domain are ...
Comments
Information & Contributors
Information
Published In
July 2024
1657 pages
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
GECCO '24
Sponsor:
GECCO '24: Genetic and Evolutionary Computation Conference
July 14 - 18, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 188Total Downloads
- Downloads (Last 12 months)188
- Downloads (Last 6 weeks)64
Reflects downloads up to 17 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in