Natural Language Programming in Medicine: Administering Evidence Based Clinical Workflows with Autonomous Agents Powered by Generative Large Language Models
Authors:
Akhil Vaid,
Joshua Lampert,
Juhee Lee,
Ashwin Sawant,
Donald Apakama,
Ankit Sakhuja,
Ali Soroush,
Sarah Bick,
Ethan Abbott,
Hernando Gomez,
Michael Hadley,
Denise Lee,
Isotta Landi,
Son Q Duong,
Nicole Bussola,
Ismail Nabeel,
Silke Muehlstedt,
Silke Muehlstedt,
Robert Freeman,
Patricia Kovatch,
Brendan Carr,
Fei Wang,
Benjamin Glicksberg,
Edgar Argulian,
Stamatios Lerakis
, et al. (5 additional authors not shown)
Abstract:
Generative Large Language Models (LLMs) hold significant promise in healthcare, demonstrating capabilities such as passing medical licensing exams and providing clinical knowledge. However, their current use as information retrieval tools is limited by challenges like data staleness, resource demands, and occasional generation of incorrect information. This study assessed the potential of LLMs to…
▽ More
Generative Large Language Models (LLMs) hold significant promise in healthcare, demonstrating capabilities such as passing medical licensing exams and providing clinical knowledge. However, their current use as information retrieval tools is limited by challenges like data staleness, resource demands, and occasional generation of incorrect information. This study assessed the potential of LLMs to function as autonomous agents in a simulated tertiary care medical center, using real-world clinical cases across multiple specialties. Both proprietary and open-source LLMs were evaluated, with Retrieval Augmented Generation (RAG) enhancing contextual relevance. Proprietary models, particularly GPT-4, generally outperformed open-source models, showing improved guideline adherence and more accurate responses with RAG. The manual evaluation by expert clinicians was crucial in validating models' outputs, underscoring the importance of human oversight in LLM operation. Further, the study emphasizes Natural Language Programming (NLP) as the appropriate paradigm for modifying model behavior, allowing for precise adjustments through tailored prompts and real-world interactions. This approach highlights the potential of LLMs to significantly enhance and supplement clinical decision-making, while also emphasizing the value of continuous expert involvement and the flexibility of NLP to ensure their reliability and effectiveness in healthcare settings.
△ Less
Submitted 22 August, 2024; v1 submitted 5 January, 2024;
originally announced January 2024.
