EP279 - Feasibility of Large Language Models in the Generation of Colorectal Surgery Board Questions

Image or Table (Do Not add text here):

Purpose/Background: Large language models (LLMs) have become increasingly integrated in healthcare. While OpenAI’s LLM chatbot ChatGPT has demonstrated proficiency by passing the United States Medical Licensing Exam, its value as a study tool for specialty board exams remains unknown. This study aims to assess whether LLMs can generate colorectal surgery multiple-choice board-style questions with minimal prompting. We hypothesize that LLMs can produce straightforward and relevant questions but will lack the depth and complexity of those found on the board exam.

Methods/Interventions: ChatGPT 4o, Google Gemini Advanced, and Microsoft Copilot Pro were instructed to create colorectal surgery multiple-choice board-style questions based on the principles discussed in the ABCRS Instructions for Question Writers document published in March 2024. A total of twenty questions, four from each of the five educational pillars (anorectal disease, benign disease, malignancy, pelvic floor, and perioperative) as delineated by the Colon and Rectal Surgery Educational Program (CARSEP), was generated by each LLM. Twenty questions from the CARSEP question bank were also included for comparison. Each question was evaluated for clarity, relevance, suitability based on difficulty, quality of distractors, and adequacy of rationale on a 5-point Likert scale (very good (5) to very poor (1)) by board-certified colorectal surgeons. Reviewers then graded questions as “Approved for Committee”, “Author to Review”, or “Not Accepted”. Reviewers were blinded to the question source. Scores were summarized as Likert means and intergroup differences were assessed by ANOVA and post-hoc analysis after variance testing by Levene’s Test.

Results/Outcomes: CARSEP questions achieved the highest percentage of “Approved for Committee” ratings at 65%, while ChatGPT 4o had the lowest at 7% (p < 0.001, Figure 1). Other LLMs performed similarly poorly with only 10% of questions approved by both Copilot Pro and Gemini. 90% or more of the questions generated by AI were either graded “Author to Review” or “Not Accepted.” ANOVA results revealed significant differences between AI performance across all measures of clarity (p=0.01), suitability (p=0.02), relevance, quality of distractors, and adequacy of rationale (all others p< 0.001). CARSEP significantly outperformed both Copilot Pro (p=0.03) and Gemini Advanced (p=0.03) but not ChatGPT 4o (p=0.50) for question correctness. For clarity, quality of distractors, and quality of rationale, CARSEP also significantly outperformed the tested AI models (p < 0.05). There was moderate correlation between the reviewers (Kendall’s tau =0.056, p< 0.001).

Conclusion/Discussion: All LLMs performed poorly compared to CARSEP, with ChatGPT 4o scoring the lowest mean Likert scores amongst the tested models. Most AI-generated questions do not meet the standard of CARSEP and at this juncture, LLMs do not appear to be a suitable resource for colorectal surgery board questions.