Skip to main content
2025 ASCRS Annual Scientific Meeting Main banner
Final Program


Visit ASCRS U
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Presentation Icons
CME
CME
Invitation/Ticketed Event
Invitation/Ticketed Event
Invited Lectureship
Invited Lectureship
Award Recipient
Award Recipient
Self-Assessment
Self-Assessment
Social
Social
Industry
Industry
Back

Session: ePoster: Quality and Cost 4 (Mon8)

EP649 - Accuracy and Reliability of Colorectal Cancer and IBD FAQs: A Comparative Analysis of Online Sources and Generative Language Model Responses

Monday, May 12, 2025
8:10 AM – 8:15 AM PT
Location: Monitor 7, Sails Pavilion, Upper Level

    ePoster Presenter(s)

  • Pia Canal Zarate, MD (she/her/hers)

    Research fellow
    Bieth Israel Deaconess Medical Center
    Boston, MA, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

    • Author(s)

  • Ana Sofia Ore, MD MSc MPH (she/her/hers)

    General Surgery Resident
    Beth Israel Deaconess Medical Center/Harvard Medical School
    Beth Israel Deaconess Medical Center/Harvard Medical School
    Boston, MA, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • VY

    Victoria Yuan, MD

    General Surgery Resident
    New York Presbytarian Queens
    NYC, New York, United States

    Disclosure information not submitted.

  • GJ

    Gary Jean, MD

    Resident
    New York Presbyterian
    Flushing, New York, United States

    Disclosure information not submitted.

  • SK

    Shahenda Khedr, BA

    Research
    New York Presbytarian Queens
    New York, New York, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • CF

    Christopher Foglia, MD

    Assistant Professor
    Weill Cornell School of Medicine
    New York, New York, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • DW

    Daniel Wong, MD

    Colorectal Surgeon
    Beth Israel Deaconess Medical Center
    Boston, Massachusetts, United States

    Disclosure information not submitted.

  • RP

    Roger Patron-Lozano, MD

    Assistant Professor
    Weill Cornell Medicine
    Flushing, New York, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • Mehraneh D. Jafari, MD (she/her/hers)

    Director of Clinical Research; Department of Surgery
    Weill Cornell Medicine
    Weill Cornell Medicine
    New York, New York, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no relevant relationships with ineligible companies.

  • Evangelos Messaris, MD, PhD, MBA

    Associate Professor of Surgery
    Division of Colon and Rectal Surgery, Beth Isreal Deaconess Medical Center
    Boston, Massachusetts, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

    • Senior Author(s)

  • SC

    Steven Y. Chao, MD FACS

    Assistant Professor of Clinical Surgery
    Weill Cornell Medicine
    Flushing, New York, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

All of the relevant financial relationships listed below have been mitigated.
Pia Canal Zarate, MD: This individual has no financial relationships with ineligible companies.
Ana Sofia Ore, MD MSc MPH: This individual has no financial relationships with ineligible companies.
Shahenda Khedr, BA: This individual has no financial relationships with ineligible companies.
Christopher Foglia, MD: This individual has no financial relationships with ineligible companies.
Roger Patron-Lozano, MD: This individual has no financial relationships with ineligible companies.
Mehraneh D. Jafari, MD: This individual has no relevant relationships with ineligible companies.
Evangelos Messaris, MD, PhD, MBA: This individual has no financial relationships with ineligible companies.
Steven Y. Chao, MD FACS: This individual has no financial relationships with ineligible companies.
Image or Table (Do Not add text here):

Purpose/Background: With the increasing availability of generative language models (GLMs) and online content, patients now face the challenge of navigating multiple sources for their medical information. While GLMs provide quick answers to most questions, our goal is to compare the accuracy and quality of responses to frequently asked questions (FAQs) in colorectal surgery with those answers found in trusted online resources.

Methods/Interventions: A qualitative survey was conducted, eleven frequently asked questions (FAQs) along with their answers regarding colorectal cancer (CRC) and inflammatory bowel disease (IBD) were selected from the ASCRS and Crohn’s & Colitis Foundation websites, answers to these questions were generated from three GLMs (Gemini, ChatGPT 3.5, and Copilot). Five board certified colorectal surgeons from three Institutions were blinded to the content source and asked to rate each response for accuracy and reliability on a five-point Likert scale. A mean score was calculated for each response. As a secondary outcome, all responses were evaluated for readability using Flesch-Kincaid grade level.

Results/Outcomes: Anonymous, blinded faculty responses were analyzed, and a mean score (1-5) was calculated for each source. For both colorectal cancer and IBD FAQs, the highest accuracy and reliability score was awarded to ChatGPT, with and overall mean score of 3.7. This was followed by website content, with a mean score of 3.5 for ASCRS and 3.3 for the Crohn’s & Colitis Foundation. The lowest-rated source overall was Gemini with a mean overall score of 3.2. ChatGPT’s responses required an 11th-grade reading level for colorectal cancer content and a freshman college level for IBD answers. In contrast, ASCRS content required a 12th-grade reading level, while the Crohn’s & Colitis Foundation content required a 10th-grade level (Table 1).

Conclusion/Discussion: Quality assessment measure performed by board certified colorectal surgeons of currently available online content, including widely used GLMs, found ChatGPT to have consistently higher ratings, surpassing the original websites from which the FAQs originated. However, all sources were found to require an advanced high school reading level, which may limit comprehension for many patients seeking online information