Skip to main content
2025 ASCRS Annual Scientific Meeting Main banner
Final Program


Visit ASCRS U
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Presentation Icons
CME
CME
Invitation/Ticketed Event
Invitation/Ticketed Event
Invited Lectureship
Invited Lectureship
Award Recipient
Award Recipient
Self-Assessment
Self-Assessment
Social
Social
Industry
Industry
Back

Session: ePoster: Education, Training Assessment, and Practice Management 2 (Mon1015)

EP720 - Are ChatGPT Answers to Patient Questions Regarding Fecal Incontinence Accurate, Complete, And Consistent with Clinical Practice Guidelines?

Monday, May 12, 2025
10:31 AM – 10:36 AM PT

    ePoster Presenter(s)

  • Lindsay Welton, MD (she/her/hers)

    General Surgery Resident
    University of Minnesota
    University of Minnesota
    Minneapolis, Minnesota, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

    • Author(s)

  • MG

    Melissa A. Gunderson, Dr.

    Surgery Resident
    University of Minnesota
    Minneapolis, Minnesota, United States

    Disclosure information not submitted.

  • GR

    Geetanjali Rajamani, BS

    Medical Student
    University of Minnesota Medical School
    Minneapolis, Minnesota, United States

    Disclosure information not submitted.

  • Karen Trang, MD

    Surgery Resident
    UCSF
    UCSF
    San Francisco, California, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • Lauren Weaver, MD

    Resident
    University of Minnesota
    University of Minnesota
    Minneapolis, MN, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • Alexander M. Troester, MD (he/him/his)

    General Surgery Resident
    University of Minnesota
    University of Minnesota
    Minneapolis, MN, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • YL

    Ying Liu, PhD

    Research Scientist
    Medical School Department of Surgery, University of Minnesota
    Minnepolis, Minnesota, United States

    Disclosure information not submitted.

  • RZ

    Rui Zhang, PhD

    Professor and Division Chief of Surgery
    University of Minnesota
    Minneapolis, Minnesota, United States

    Disclosure information not submitted.

  • MW

    Mark L. Welton, MD, MHCM

    Professor of Surgery
    Univeristy of Minnesota
    Minneapolis, Minnesota, United States

    Disclosure information not submitted.

  • Elizabeth C. Wick, MD

    Professor of Surgery
    University of California, San Francisco
    University of California, San Francisco
    San Francisco, California, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

    • Senior Author(s)

  • GM

    Genevieve B. Melton, MD, PhD

    Professor of Surgery
    Univeristy of Minnesota
    Minneapolis, Minnesota, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

All of the relevant financial relationships listed below have been mitigated.
Lindsay Welton, MD: This individual has no financial relationships with ineligible companies.
Karen Trang, MD: This individual has no financial relationships with ineligible companies.
Lauren Weaver, MD: This individual has no financial relationships with ineligible companies.
Alexander M. Troester, MD: This individual has no financial relationships with ineligible companies.
Elizabeth C. Wick, MD: This individual has no financial relationships with ineligible companies.
Genevieve B. Melton, MD, PhD: This individual has no financial relationships with ineligible companies.
Image or Table (Do Not add text here):

Purpose/Background: Large language models (LLMs) such as Chat Generative Pre-Trained Transformer (ChatGPT) are capable of passing professional medical examinations and are revolutionizing how patients obtain information, including for sensitive health issues. This study evaluated the accuracy, completeness, and consistency of ChatGPT-3.5 and ChatGPT-4’s answers to questions regarding fecal incontinence (FI). We hypothesized neither ChatGPT version would provide reliable information.

Methods/Interventions: Thirty lay language questions were developed based on the American Society of Colon and Rectal Surgeons Clinical Practice Guideline (ASCRS-CPG) for FI. Content fell into one of four categories: background, diagnosis, treatment, or miscellaneous. Questions were presented twice to both ChatGPT versions with temperature set to 0. The chat was cleared between rounds, yielding four sets of responses. Three professors of colorectal surgery (CRS) rated the answers “yes” or “no” for accuracy (contained only correct information), completeness (was comprehensive), and consistency (concordant with ASCRS-CPG recommendations). A “No” prompted a free text explanation. T-tests assessed similarity of answers between rounds. Cohen’s Kappa evaluated interrater agreement. Descriptive statistics were completed. Rapid qualitative analysis on reviewer comments by a priori themes (accuracy, completeness, and consistency) and emergent sub-themes were conducted with NVivo.

Results/Outcomes: Reviewers rated 61% of answers accurate, 65% complete, and 68% consistent for ChatGPT-3.5, and 72%, 73%, and 69% for ChatGPT-4, respectively. Together, 3.5 and 4 did best on miscellaneous questions, averaging 83% across accuracy, completeness, and consistency, and worst on treatment questions (64%). ChatGPT-3.5 provided different answers between rounds for background and treatment questions (p-values 0.01), whereas ChatGPT-4 had variability among background questions only (p=0.03). Two miscellaneous questions: “Is it bad to have FI?” and “Can anal sex cause FI?” variably triggered ChatGPT’s content warning, flagging them as inappropriate and terminating the chat. Qualitative analyses showed the most frequent theme identified in reviewer responses was inaccuracy for “treatment recommendations,” with a total of 105 reviewer annotations such as “temperature-controlled radiofrequency energy to the sphincter complex is NOT recommended to treat FI.” The next most common theme was inaccuracy regarding “physiology” (91 annotations), with comments including: “urgency is not the same as incontinence.”

Conclusion/Discussion: Neither version of ChatGPT provided answers that were accurate, complete, or consistent with the ASCRS-CPG for FI. Quantitative and qualitative analysis showed both versions performed particularly poorly regarding treatment recommendations, averaging 64% or a “D,” often suggesting treatments that explicitly contradicted guidelines. Additionally, flagging sensitive health questions as “inappropriate” risks isolating patients by further stigmatizing their disease. This study emphasizes the necessity for providers to educate patients on pitfalls associated with this technology, actively participate in the integration of health information and LLMs to ensure up-to-date information is available to patients, and work as advocates to combat stigma around this prevalent and important colorectal condition.