Abstract
Operational decisions such as loan or subsidy allocation are taken with high frequency and require a consistent decision quality which decision models can ensure. Decision models can be derived from textual descriptions describing both the decision logic and decision dependencies. Whilst decision models already help with modeling, implementing and automating decisions, the modelling step would still benefit from a (semi)-automated approach. The introduction of ChatGPT and GPT-3 offers opportunities to automatically discover decision dependencies from a given text. This paper evaluates the performance of two approaches that automatically extract decision dependencies from text, namely the best performing version of GPT-3 with a BERT-based approach. An evaluation with 36 experiments with a dataset of real-life cases and various levels of creativity allowed for GPT-3 concludes that theBERT BERT-based approach outperforms GPT-3 on the real-life dataset but that GPT-3 has promising results and requires further investigation.
This work was supported by the Fund for Scientific Research Flanders (project G079519N) and KU Leuven Internal Funds (project C14/19/082).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arco, L., Nápoles, G., Vanhoenshoven, F., Lara, A.L., Casas, G., Vanhoof, K.: Natural language techniques supporting decision modelers. Data Min. Knowl. Disc. 35(1), 290–320 (2021)
Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
Busch, K., Rochlitzer, A., Sola, D., Leopold, H.: Just tell me: prompt engineering in business process management. In: van der Aa, H., Bork, D., Proper, H.A., Schmidt, R. (eds.) BPMDS EMMSAD 2023. LNBIP, vol. 479, pp. 3–11. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-34241-7_1
Clark, K., Manning, C.D.: Deep reinforcement learning for mention-ranking coreference models. arXiv preprint arXiv:1609.08667 (2016)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dziri, N., et al.: Faith and fate: limits of transformers on compositionality. arXiv preprint arXiv:2305.18654 (2023)
Etikala, V., Van Veldhoven, Z., Vanthienen, J.: Text2Dec: extracting decision dependencies from natural language text for automated DMN decision modelling. In: Del Río Ortega, A., Leopold, H., Santoro, F.M. (eds.) BPM 2020. LNBIP, vol. 397, pp. 367–379. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-66498-5_27
Fill, H.G., Fettke, P., Köpke, J.: Conceptual modeling and large language models: impressions from first experiments With ChatGPT. Enterp. Modell. Inf. Syst. Archit. (EMISAJ) Int. J. Conceptual Model. 18, 1–15 (2023). https://doi.org/10.18417/emisa.18.3
Goossens, A., De Smedt, J., Vanthienen, J.: Extracting decision model and notation models from text using deep learning techniques. Expert Syst. Appl. 211, 118667 (2023)
Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
Hwang, J.D., et al.: (Comet-) atomic 2020: on symbolic and neural commonsense knowledge graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 6384–6392 (2021)
Jojic, A., Wang, Z., Jojic, N.: GPT is becoming a turing machine: here are some ways to program it. arXiv preprint arXiv:2303.14310 (2023)
Khurana, D., Koli, A., Khatter, K., Singh, S.: Natural language processing: state of the art, current trends and challenges. Multimed. Tools Appl. 1–32 (2022)
Kluza, K., Honkisz, K.: From SBVR to BPMN and DMN models. Proposal of translation from rules to process and decision models. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2016. LNCS (LNAI), vol. 9693, pp. 453–462. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39384-1_39
Liu, R., Mao, R., Luu, A.T., Cambria, E.: A brief survey on recent advances in coreference resolution. Artif. Intell. Rev. 1–43 (2023)
OMG: Omg: Decision model and notation 1.0 (2015) (2008). https://www.omg.org/spec/DMN/1.0/. Accessed 08 Jan 2022
OMG: Omg: Semantics of business vocabulary and rules (2008) (2008). https://www.omg.org/spec/SBVR/1.0. Accessed 13 Mar 2023
Quishpi, L., Carmona, J., Padró, L.: Extracting decision models from textual descriptions of processes. In: Polyvyanyy, A., Wynn, M.T., Van Looy, A., Reichert, M. (eds.) BPM 2021. LNCS, vol. 12875, pp. 85–102. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85469-0_8
Vanthienen, J.: Decisions, advice and explanation: an overview and research agenda. In: A Research Agenda for Knowledge Management and Analytics, pp. 149–169. Edward Elgar Publishing (2021)
Vanthienen, J., Mues, C., Aerts, A.: An illustration of verification and validation in the modelling phase of KBS development. Data Knowl. Eng. 27(3), 337–352 (1998)
Wei, J., et al.: Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)
Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Goossens, A., De Smedt, J., Vanthienen, J. (2024). Comparing the Performance of GPT-3 with BERT for Decision Requirements Modeling. In: Sellami, M., Vidal, ME., van Dongen, B., Gaaloul, W., Panetto, H. (eds) Cooperative Information Systems. CoopIS 2023. Lecture Notes in Computer Science, vol 14353. Springer, Cham. https://doi.org/10.1007/978-3-031-46846-9_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-46846-9_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46845-2
Online ISBN: 978-3-031-46846-9
eBook Packages: Computer ScienceComputer Science (R0)