Skip to main content
2025 ASCRS Annual Scientific Meeting Main banner
Final Program


Visit ASCRS U
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Presentation Icons
CME
CME
Invitation/Ticketed Event
Invitation/Ticketed Event
Invited Lectureship
Invited Lectureship
Award Recipient
Award Recipient
Self-Assessment
Self-Assessment
Social
Social
Industry
Industry
Back

Session: Quick Shot: Rectal Cancer

QS131 - Evaluating the Accuracy of ChatGPT-4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal Adenocarcinoma

Sunday, May 11, 2025
3:55 PM – 3:58 PM PT
Location: Room 9, Upper Level

    Quick Shot Presenter(s)

  • Ryan Meyer, MD (he/him/his)

    General Surgery Resident
    Los Robles Regional Medical Center
    Thousand Oaks, CA, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

    • Author(s)

  • KP

    Kevin Palmer, M.D.

    General Surgery Resident
    Los Robles Medical Center
    Thousand Oaks, California, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • SP

    Shivam Pandya, MD

    Resident Physician
    Los Robles Regional Medical Center Department of Surgery
    Thousand Oaks, California, United States

    Disclosure information not submitted.

  • TB

    Tamir E. Bresler, MD

    General Surgery PGY-3
    Los Robles Regional Medical Center
    Thousand Oaks, California, United States

    Disclosure information not submitted.

    • Senior Author(s)

  • MF

    Manabu Fujita, M.D.

    Surgical Oncologist
    Los Robles Medical Center
    Thousand Oaks, California, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

All of the relevant financial relationships listed below have been mitigated.
Ryan Meyer, M.D.: This individual has no financial relationships with ineligible companies.
Kevin Palmer, M.D.: This individual has no financial relationships with ineligible companies.
Manabu Fujita, M.D.: This individual has no financial relationships with ineligible companies.
Image or Table (Do Not add text here):

Purpose/Background: Artificial intelligence (AI) continues to play an increasingly important role in modern health care. ChatGPT (OpenAI, San Francisco, CA) has previously shown potential in its utility to assist with medical decision making. It’s use in complicated clinical scenarios that commonly occur in cases of rectal adenocarcinoma has not been investigated. The objective of this study is to explore the role of ChatGPT-4o in guiding clinical decision-making in rectal adenocarcinoma.

Methods/Interventions: We reviewed the National Comprehensive Cancer Network® (NCCN) Clinical Practice Guidelines for the Management of Rectal Adenocarcinoma (Version 3.2024) and formulated three complex clinical questions for each decision-making page. ChatGPT-4o was queried in a reproducible fashion by 2 physicians. We scored answers on the following Likert scale: 5) Correct; 4) Correct, with missing information requiring clarification; 3) Correct, but unable to complete answer; 2) Partially incorrect; 1) Absolutely incorrect. Score frequencies were compared, and subgroup analysis was conducted on Correctness (defined as scores 1-2 vs 3-5) and Accuracy (scores 1-3 vs 4-5). The questions were grouped into several categories including workup, treatment, surveillance, and diagnostics. A subgroup analysis was conducted using the Kruskal-Wallis test to assess the distribution of scores across these categories.

Results/Outcomes: In total, 45-pages of the NCCN Guidelines® were analyzed, generating 135 complex clinical questions that were evaluated by 2 reviewers. The percentage of responses considered Correct was 89.6%. The percentage of responses considered Accurate was 94.1%. The subgroup analysis using the Kruskal-Wallis test showed no statistically significant difference in scoring distribution among the four intervention types (H= 7.27, p= 0.12).

Conclusion/Discussion: ChatGPT represents an interesting and increasingly accurate tool. AI has potential use in clinical decision making in the near future if the AI error rates become lower than human error rates. Its application in real-time decision making, such as in a multidisciplinary tumor boards, could offer significant utility. Future research is needed to determine the validity of this tool for clinical applications.