Large Language Models for the Automated Analysis of Optimization Algorithms
Pages 160 - 168
Abstract
The ability of Large Language Models (LLMs) to generate high-quality text and code has fuelled their rise in popularity. In this paper, we aim to demonstrate the potential of LLMs within the realm of optimization algorithms by integrating them into STNWeb. This is a web-based tool for the generation of Search Trajectory Networks (STNs), which are visualizations of optimization algorithm behavior. Although visualizations produced by STNWeb can be very informative for algorithm designers, they often require a certain level of prior knowledge to be interpreted. In an attempt to bridge this knowledge gap, we have incorporated LLMs, specifically GPT-4, into STNWeb to produce extensive written reports, complemented by automatically generated plots, thereby enhancing the user experience and reducing the barriers to the adoption of this tool by the research community. Moreover, our approach can be expanded to other tools from the optimization community, showcasing the versatility and potential of LLMs in this field.
References
[1]
Christian Blum and Andrea Roli. 2003. Metaheuristics in combinatorial optimization: Overview and conceptual comparison. Comput. Surveys 35, 3 (2003), 268--308.
[2]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. 2020. Language Models are Few-Shot Learners. arXiv:cs.CL/2005.14165
[3]
Camilo Chacon-Sartori, Christian Blum, and Gabriela Ochoa. 2023. Search Trajectory Networks Meet the Web: A Web Application for the Visual Comparison of Optimization Algorithms. In Proceedings of the 2023 12th International Conference on Software and Computer Applications (ICSCA '23). Association for Computing Machinery, New York, NY, USA, 89--96.
[4]
Camilo Chacón Sartori, Christian Blum, and Gabriela Ochoa. 2023. STNWeb: A new visualization tool for analyzing optimization algorithms. Software Impacts 17 (2023), 100558.
[5]
Philip Feldman, James R. Foulds, and Shimei Pan. 2023. Trapping LLM Hallucinations Using Tagged Context Prompts. arXiv:cs.CL/2306.06085
[6]
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, et al. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:cs.CL/2312.10997
[7]
Michel Gendreau and Jean-Yves Potvin. 2005. Metaheuristics in combinatorial optimization. Annals of Operations Research 140, 1 (2005), 189--213.
[8]
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, et al. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv:cs.CL/2311.05232
[9]
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, et al. 2023. Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. arXiv:cs.CL/2311.10702
[10]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al. 2023. Mistral 7B. arXiv:cs.CL/2310.06825
[11]
Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. 2023. Evaluating Open-Domain Question Answering in the Era of Large Language Models. arXiv:cs.CL/2305.06984
[12]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. arXiv:cs.CL/2205.11916
[13]
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 3045--3059.
[14]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, et al. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:cs.CL/2005.11401
[15]
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, et al. 2022. Solving Quantitative Reasoning Problems with Language Models. arXiv:cs.CL/2206.14858
[16]
Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, et al. 2023. Guiding Large Language Models via Directional Stimulus Prompting. arXiv:cs.CL/2302.11520
[17]
Yen-Ting Lin and Yun-Nung Chen. 2023. LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models. arXiv:cs.CL/2305.13711
[18]
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, et al. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv:cs.CL/2107.13586
[19]
Paula Maddigan and Teo Susnjak. 2023. Chat2VIS: Fine-Tuning Data Visualisations using Multilingual Natural Language Text and Pre-Tramed Large Language Models. arXiv:cs.HC/2303.14292
[20]
Paula Maddigan and Teo Susnjak. 2023. Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models. IEEE Access 11 (2023), 45181--45193.
[21]
Gabriela Ochoa, Katherine M. Malan, and Christian Blum. 2020. Search trajectory networks of population-based algorithms in continuous spaces. In Proceedings of EvoApps 2020 - International Conference on the Applications of Evolutionary Computation. Springer, 70--85.
[22]
Gabriela Ochoa, Katherine M. Malan, and Christian Blum. 2021. Search trajectory networks: A tool for analysing and visualising the behaviour of metaheuristics. Applied Soft Computing 109 (2021), 107492.
[23]
OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2023. GPT-4 Technical Report. arXiv:cs.CL/2303.08774
[24]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, et al. 2022. Training language models to follow instructions with human feedback. arXiv:cs.CL/2203.02155
[25]
Michal Pluhacek, Anezka Kazikova, Tomas Kadavy, Adam Viktorin, and Roman Senkerik. 2023. Leveraging Large Language Models for the Generation of Novel Metaheuristic Optimization Algorithms. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation (GECCO '23 Companion). Association for Computing Machinery, New York, NY, USA, 1812--1820.
[26]
XiPeng Qiu, TianXiang Sun, YiGe Xu, YunFan Shao, Ning Dai, et al. 2020. Pretrained models for natural language processing: A survey. Science China Technological Sciences 63, 10 (Sept. 2020), 1872--1897.
[27]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, et al. 2021. Zero-Shot Text-to-Image Generation. arXiv:cs.CV/2102.12092
[28]
Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. arXiv [cs.CL] (Feb. 2021).
[29]
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, et al. 2023. Code Llama: Open foundation models for code. arXiv [cs.CL] (Aug. 2023).
[30]
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. arXiv:cs.LG/2110.08207
[31]
Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. 2023. Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL. arXiv:cs.CL/2309.06553
[32]
Jiao Sun, Q Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, et al. 2022. Investigating explainability of generative AI for code through scenario-based design. In 27th International Conference on Intelligent User Interfaces. ACM, New York, NY, USA.
[33]
Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, et al. 2023. Text Classification via Large Language Models. arXiv:cs.CL/2305.08377
[34]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, et al. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv:cs.CL/2312.11805
[35]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:cs.CL/2307.09288
[36]
Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHIEA '22). Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages
[37]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, et al. 2023. Attention Is All You Need. arXiv:cs.CL/1706.03762
[38]
Albert Webson and Ellie Pavlick. 2021. Do Prompt-Based Models Really Understand the Meaning of their Prompts? CoRR abs/2109.01247 (2021). arXiv:2109.01247 https://arxiv.org/abs/2109.01247
[39]
Jason Wei, Yi Tay, Risbi Bommasani, Colin Raffel, Barret Zoph, et al. 2022. Emergent Abilities of Large Language Models. arXiv:cs.CL/2206.07682
[40]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, et al. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:cs.CL/2201.11903
[41]
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, et al. 2023. A Survey on Multimodal Large Language Models. arXiv:cs.CV/2306.13549
[42]
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting Large Language Model for Machine Translation: A Case Study. arXiv:cs.CL/2301.07069
[43]
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023. Sentiment Analysis in the Era of Large Language Models: A Reality Check. arXiv:cs.CL/2305.15005
[44]
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, et al. 2023. Explainability for Large Language Models: A Survey. arXiv:cs.CL/2309.01029
[45]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:cs.CL/2306.05685
[46]
Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, et al. 2024. A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends. arXiv:cs.SE/2311.10372
[47]
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, et al. 2022. Large language models are human-level prompt engineers. arXiv [cs.LG] (Nov. 2022).
[48]
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Ziehen Wang, Hao Chen, et al. 2023. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. arXiv:cs.CL/2306.04528
Index Terms
- Large Language Models for the Automated Analysis of Optimization Algorithms
Recommendations
Leveraging Large Language Models for the Generation of Novel Metaheuristic Optimization Algorithms
GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary ComputationIn this paper, we investigate the potential of using Large Language Models (LLMs) such as GPT-4 to generate novel hybrid swarm intelligence optimization algorithms. We use the LLM to identify and decompose six well-performing swarm algorithms for ...
A theoretical and empirical analysis of convergence related particle swarm optimization
In this paper an extensive theoretical and empirical analysis of recently introduced Particle Swarm Optimization algorithm with Convergence Related parameters (CR-PSO) is presented. The convergence of the classical PSO algorithm is addressed in detail. ...
Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study
SIGMODThe Natural Language to Visualization (NL2Vis) task aims to transform natural-language descriptions into visual representations for a grounded table, enabling users to gain insights from vast amounts of data. Recently, many deep learning-based approaches ...
Comments
Information & Contributors
Information
Published In
July 2024
1657 pages
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
GECCO '24
Sponsor:
GECCO '24: Genetic and Evolutionary Computation Conference
July 14 - 18, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 107Total Downloads
- Downloads (Last 12 months)107
- Downloads (Last 6 weeks)14
Reflects downloads up to 24 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in