EP524 - Creating Consensus Statements For Ventral Rectopexy: Three Large Language Models Versus International Experts

Image or Table (Do Not add text here):

Purpose/Background: Physicians and scientists are increasingly turning to large language models (LLMs) to help with research, clinical writing, and decision making. However, the understanding of where LLMs source medical information, how they synthesize it, and the quality of their output is limited. In 2024, an expert group of Colon and Rectal surgeons created an international consensus on ventral rectopexy (VR) comprising of 17 statements on preoperative assessment, surgical technique, and outcomes. This study set out to test LLMs' ability to curate literature and synthesize information on VR prior to these statements’ publication.

Methods/Interventions: Seventeen primary LLM prompts were derived from the 17 expert consensus (EC) statements. Secondary prompts were derived from key discussion topics from within each statement’s supporting text. Primary prompts were presented to three commonly used LLMs: ChatGPT 4o, Gemini Advanced, and OpenEvidence. The secondary prompts were sequentially presented if the key topic had not been generated from prior prompts. Responses were then assessed for completeness of content, word count, citation quantity, citation quality (e.g., PubMed indexed vs Wikipedia), and citation accuracy (e.g., an incorrect citation traced to an existing source, vs entirely fabricated citation).

Results/Outcomes: ChatGPT reproduced all EC content in 4 of 17 statements (24%) and all but one detail in 3 statements. Gemini reproduced all content in 6 statements (35%) and all but one detail in 3 statements. OpenEvidence reproduced all content in 7 statements (41%) and all but one detail in 5 statements. There was no statistical difference between LLMs (p = 0.26). While the models synthesized basic indications and clinical evaluation questions well, all struggled to gather relevant information on complex topics including male VR and VR in pregnancy. The number of prompts required to produce key topics in all 17 EC statements was not significantly different between models: 44 prompts for ChatGPT, 58 for Gemini, and 61 for OpenEvidence.

ChatGPT produced 167 citations of 77 unique sources; 48.5% of citations were PubMed indexed, and 50.9% were fabricated. Among non-fabricated sources, ChatGPT provided misleading source information such as fabricated authorship, title, or journal name 65.1% of the time. Gemini produced 248 citations of 210 unique sources; 48.0% were PubMed indexed, 20.6% were non-peer reviewed medical sources (e.g. hospital websites), 18.1% were nonmedical sources (e.g. Wikipedia), and 13.3% were fabricated. OpenEvidence produced 171 citations of 123 unique PubMed indexed sources with no fabrications. None of the LLMs chose more than 13 (12.3%) of the 106 EC citations (Table 1).

All LLMs were significantly more verbose than experts. Chat GPT had an average response of 1669 words per consensus statement, Gemini 968 words, and OpenEvidence 808 words; the experts averaged only 336 words per statement (p < 0.01).

Conclusion/Discussion: When provided with detailed prompts, LLMs produced lengthy statements regarding VR, but frequently failed to synthesize information deemed important by expert consensus or to cite expert-selected sources. ChatGPT and Gemini fabricated citations. While OpenEvidence may be a valuable tool for literature search, no LLM appears ready to synthesize diagnostic or treatment information when this information is not already explicitly concatenated in the literature.