Skip to main content
2025 ASCRS Annual Scientific Meeting Main banner
Final Program


Visit ASCRS U
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Presentation Icons
CME
CME
Invitation/Ticketed Event
Invitation/Ticketed Event
Invited Lectureship
Invited Lectureship
Award Recipient
Award Recipient
Self-Assessment
Self-Assessment
Social
Social
Industry
Industry
Back

Session: ePoster: Education, Training Assessment, and Practice Management 3 (Tue8)

EP949 - Benchmarking Artificial Intelligence Performance on Education of Benign Anorectal Conditions

Tuesday, May 13, 2025
8:10 AM – 8:15 AM PT

    ePoster Presenter(s)

  • MM

    Makenna Marty, MD

    General Surgery Resident
    Huntington Hospital
    Pasadena, CA, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

    • Author(s)

  • SM

    Seija Maniskas, MD

    General Surgery Resident
    Huntington Hospital
    Pasadena, California, United States

    Disclosure information not submitted.

  • Jonathan Zuo, MD

    General Surgery Resident
    Huntington Hospital

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • JG

    Juliane Y. Golan, MD

    Attending
    Huntington Hospital
    Pasadena, California, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

  • GA

    Gabriel Akopian, MD, MBS

    General Surgery Residency Program Director
    Huntington Hospital
    Pasadena, California, United States

    Disclosure information not submitted.

    • Senior Author(s)

  • AT

    Adam Truong, MD, MS

    Attending
    Huntington Hospital
    pasadena, California, United States

    All of the relevant financial relationships listed below have been mitigated.
    This individual has no financial relationships with ineligible companies.

All of the relevant financial relationships listed below have been mitigated.
Makenna Marty, MD: This individual has no financial relationships with ineligible companies.
Jonathan Zuo, MD: This individual has no financial relationships with ineligible companies.
Juliane Y. Golan, MD: This individual has no financial relationships with ineligible companies.
Adam Truong, MD, MS: This individual has no financial relationships with ineligible companies.
Image or Table (Do Not add text here):

Purpose/Background: Artificial intelligence (AI), though still in early stages in healthcare applications, shows promise as a valuable resource for patients due to its broad accessibility. However, its accuracy and reliability in providing medical information is largely untested. We aim to evaluate the ability of ChatGPT, the most widely recognized AI platform, to provide patient-level information on common benign anorectal conditions. Our hypothesis is that ChatGPT can provide some accurate information, but will not be able to distinguish between reliable medical sources and disinformation.

Methods/Interventions: A prospective, comparative study was conducted evaluating ChatGPT’s responses against validated medical literature sources, including PubMed and UpToDate, on the most common benign anorectal conditions worldwide: hemorrhoids, anal fissures, and pruritus ani. Information quality and accuracy for each topic of interest was assessed using the Modified Ensuring Quality Information for Patients (EQIP) tool, a quality assessment scoring system that evaluates the reliability of written healthcare communication for patients. The Modified EQIP consists of 36 standardized items within three domains: content, identification, and structure. AI answers were recorded and categorized as “Yes” for correct and complete answers, or “No” for incorrect, incomplete, or contradictory answers or for not applicable items. The process was repeated 3 times per EQIP item per anorectal condition. All answers were independently assessed by multiple authors and aggregated.

Results/Outcomes: ChatGPT had an overall aggregate score of 61% on the Modified EQIP for all topics, all domains, and all reviewers. It consistently scored highest in the content domain for all three topics (range 61.1%-76.5%), while scoring lowest in the identification domain (range 33.3%-50.0%), where the Modified EQIP emphasizes questions about sources (Figure 1). ChatGPT performed best for hemorrhoids at 63.5% correct, and worst for pruritus ani at 55.5%. Points were largely deducted due to inclusion of non-evidence-based treatments, inability to provide reliable sources of information, generation of broken links when asked to provide its resources, and providing extraneous detail irrelevant to the posed question.

Conclusion/Discussion: ChatGPT appears to deliver mostly accurate information at an accessible reading and comprehension level. However, it demonstrated notable deficiencies, including the recommendation of treatment modalities that are not evidence-based and failure to produce reliable resources and links. Given these limitations, we advise caution before ChatGPT is trusted as a reliable source of medical information for benign anorectal conditions.