research-article

FlowMind: Automatic Workflow Generation with LLMs

Authors:

William Watson,

Shayleen Reynolds,

Manuela VelosoAuthors Info & Claims

ICAIF '23: Proceedings of the Fourth ACM International Conference on AI in Finance

Pages 73 - 81

https://doi.org/10.1145/3604237.3626908

Published: 25 November 2023 Publication History

Abstract

The rapidly evolving field of Robotic Process Automation (RPA) has made significant strides in automating repetitive processes, yet its effectiveness diminishes in scenarios requiring spontaneous or unpredictable tasks demanded by users. This paper introduces a novel approach, FlowMind, leveraging the capabilities of Large Language Models (LLMs) such as Generative Pretrained Transformer (GPT), to address this limitation and create an automatic workflow generation system. In FlowMind, we propose a generic prompt recipe for a lecture that helps ground LLM reasoning with reliable Application Programming Interfaces (APIs). With this, FlowMind not only mitigates the common issue of hallucinations in LLMs, but also eliminates direct interaction between LLMs and proprietary data or code, thus ensuring the integrity and confidentiality of information — a cornerstone in financial services. FlowMind further simplifies user interaction by presenting high-level descriptions of auto-generated workflows, enabling users to inspect and provide feedback effectively. We also introduce NCEN-QA, a new dataset in finance for benchmarking question-answering tasks from N-CEN reports on funds. We used NCEN-QA to evaluate the performance of workflows generated by FlowMind against baseline and ablation variants of FlowMind. We demonstrate the success of FlowMind, the importance of each component in the proposed lecture recipe, and the effectiveness of user interaction and feedback in FlowMind.

References

[1]

1984. Edgar. https://www.sec.gov/edgar.

[2]

2022. LangChain. https://github.com/langchain-ai/langchain.

[3]

2022. OpenAI API. https://platform.openai.com/docs/guides/embeddings.

[4]

2023. AutoGPT. https://github.com/Significant-Gravitas/Auto-GPT.

[5]

2023. Transformer Agent. https://huggingface.co/docs/transformers/main_classes/agent.

[6]

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022).

[7]

Max Bachmann. 2021. maxbachmann/RapidFuzz: Release 1.8.0. https://doi.org/10.5281/zenodo.5584996

[8]

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. PAL: Program-aided Language Models. ArXiv abs/2211.10435 (2022).

[9]

Lukas-Valentin Herm, Christian Janiesch, Alexander Helm, Florian Imgrund, Kevin Fuchs, Adrian Hofmann, and Axel Winkelmann. 2020. A Consolidated Framework for Implementing Robotic Process Automation Projects. In Business Process Management, Dirk Fahland, Chiara Ghidini, Jörg Becker, and Marlon Dumas (Eds.). Springer International Publishing, Cham, 471–488.

[10]

Peter Hofmann, Caroline Samp, and Nils Urbach. 2020. Robotic process automation. Electronic Markets 30 (2020), 99–106. https://doi.org/10.1007/s12525-019-00365-8

[11]

Toru Kobayashi, Kenichi Arai, Tetsuo Imai, Shigeaki Tanimoto, Hiroyuki Sato, and Atsushi Kanai. 2019. Communication Robot for Elderly Based on Robotic Process Automation. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. 251–256. https://doi.org/10.1109/COMPSAC.2019.10215

[12]

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 9493–9500.

[13]

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210 (2023).

[14]

Xiao-Yang Liu, Guoxuan Wang, and Daochen Zha. 2023. FinGPT: Democratizing Internet-scale Data for Financial Large Language Models. arXiv preprint arXiv:2307.10485 (2023).

[15]

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. WebGPT: Browser-assisted question-answering with human feedback. arxiv:2112.09332 [cs.CL]

[16]

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022).

[17]

Jayr Pereira, Robson Fidalgo, Roberto Lotufo, and Rodrigo Nogueira. 2023. Visconde: Multi-document QA with GPT-3 and Neural Reranking. In European Conference on Information Retrieval. Springer, 534–543.

Digital Library

[18]

Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201.11227 (2022).

[19]

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, 2018. Improving language understanding by generative pre-training. (2018).

[20]

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019).

[21]

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083 (2023).

[22]

M. Ratia, J. Myllärniemi, and N. Helander. 2018. Robotic Process Automation - Creating Value by Digitalizing Work in the Private Healthcare?. In Proceedings of the 22nd International Academic Mindtrek Conference (Tampere, Finland) (Mindtrek ’18). Association for Computing Machinery, New York, NY, USA, 222–227. https://doi.org/10.1145/3275116.3275129

Digital Library

[23]

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633 (2021).

[24]

Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools.

[25]

Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. 2023. Modular Visual Question Answering via Code Generation. arxiv:2306.05392 [cs.CL]

[26]

Rehan Syed, Suriadi Suriadi, Michael Adams, Wasana Bandara, Sander J.J. Leemans, Chun Ouyang, Arthur H.M. ter Hofstede, Inge van de Weerd, Moe Thandar Wynn, and Hajo A. Reijers. 2020. Robotic Process Automation: Contemporary themes and challenges. Computers in Industry 115 (2020), 103162. https://doi.org/10.1016/j.compind.2019.103162

Digital Library

[27]

Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts. 1–7.

Digital Library

[28]

Wil M. P. van der Aalst, Martin Bichler, and Armin Heinzl. 2018. Robotic Process Automation. Business & Information Systems Engineering 60 (2018), 269–272. https://doi.org/10.1007/s12599-018-0542-4

[29]

Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. 2023. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res 2 (2023), 20.

[30]

Alice Saldanha Villar and Nawaz Khan. 2021. Robotic process automation in banking industry: a case study on Deutsche Bank. Journal of Banking and Financial Technology 5, 1 (2021), 71–86.

[31]

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. BloombergGPT: A Large Language Model for Finance. arxiv:2303.17564 [cs.LG]

[32]

Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. FinGPT: Open-Source Financial Large Language Models. arXiv preprint arXiv:2306.06031 (2023).

[33]

Boyu Zhang, Hongyang Yang, and Xiao-Yang Liu. 2023. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models. arXiv preprint arXiv:2306.12659 (2023).

Cited By

Xu JDu WLiu XLi XFilkov VRay BZhou M(2024)LLM4Workflow: An LLM-based Automated Workflow Model Generation ToolProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695360(2394-2398)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695360
Zhang TPeiguo FLiu JZhang YChen X(2024)NLDesign: A UI Design Tool for Natural Language InterfacesProceedings of the ACM Turing Award Celebration Conference - China 202410.1145/3674399.3674455(153-158)Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3674399.3674455
Bianchini DGarda MMelchiori MRula A(2024)Leveraging Large Language Models for Data Service Discovery2024 IEEE International Conference on Web Services (ICWS)10.1109/ICWS62655.2024.00128(1097-1099)Online publication date: 7-Jul-2024
https://doi.org/10.1109/ICWS62655.2024.00128
Show More Cited By

Index Terms

FlowMind: Automatic Workflow Generation with LLMs
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
      1. Cognitive robotics
    2. Natural language processing
2. Information systems
  1. Information retrieval
    1. Users and interactive retrieval

Recommendations

Automatic workflow verification and generation

Correctness is an important aspect of workflow management systems. However, most of the workflow literature focuses only on the modeling aspects and assumes that a workflow is correct if during the execution it respects the control and data dependency ...
Automated generation of scientific workflow generators with WfChef
Abstract
Scientific workflow applications have gained significant importance, and their automated and efficient execution on large-scale computing platforms has been the subject of extensive research and development. For these efforts to be successful, a ...
Highlights
- Automatic construction of realistic synthetic workflow generators.
- Definition of metrics for quantifying the realism of generated workflows.
- Significant improvement over state-of-the-art hand-crafted generators.
- Automated ...
Automatic discovery of workflow models

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICAIF '23: Proceedings of the Fourth ACM International Conference on AI in Finance

November 2023

697 pages

ISBN:9798400702402

DOI:10.1145/3604237

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICAIF '23

ICAIF '23: 4th ACM International Conference on AI in Finance

November 27 - 29, 2023

NY, Brooklyn, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
683
Total Downloads

Downloads (Last 12 months)683
Downloads (Last 6 weeks)30

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu JDu WLiu XLi XFilkov VRay BZhou M(2024)LLM4Workflow: An LLM-based Automated Workflow Model Generation ToolProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695360(2394-2398)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695360
Zhang TPeiguo FLiu JZhang YChen X(2024)NLDesign: A UI Design Tool for Natural Language InterfacesProceedings of the ACM Turing Award Celebration Conference - China 202410.1145/3674399.3674455(153-158)Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3674399.3674455
Bianchini DGarda MMelchiori MRula A(2024)Leveraging Large Language Models for Data Service Discovery2024 IEEE International Conference on Web Services (ICWS)10.1109/ICWS62655.2024.00128(1097-1099)Online publication date: 7-Jul-2024
https://doi.org/10.1109/ICWS62655.2024.00128
Schnepf JEngin TAnderer SScheuermann B(2024)Studies on the Use of Large Language Models for the Automation of Business Processes in Enterprise Resource Planning SystemsNatural Language Processing and Information Systems10.1007/978-3-031-70239-6_2(16-31)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-70239-6_2
Casciani ABernardi MCimitile MMarrella A(2024)Conversational Systems for AI-Augmented Business Process ManagementResearch Challenges in Information Science10.1007/978-3-031-59465-6_12(183-200)Online publication date: 2-May-2024
https://doi.org/10.1007/978-3-031-59465-6_12

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents