QS106 - Utility and Bias in Standardized Letter of Recommendations for Colorectal Surgery Fellowship Applicants

Image or Table (Do Not add text here):

Purpose/Background: Letters of recommendation (LOR) play an integral role in selecting colon and rectal surgery fellowship applicants to interview and subsequently rank. To attempt to minimize bias and performance inflation, standardized LOR was implemented by the Association of Program Directors for Colon and Rectal Surgery. This study seeks to determine if there is persistent performance inflation or underlying bias in these standardized LORs.

Methods/Interventions: This is a single institution retrospective study. All letters of recommendation for the 2024 applicant cycle were included. The standardized LOR scoring system was utilized for all applicants. Letter writers are asked to rank applicants across 11 domains including timeliness in completing administrative tasks, responding to communications, commitment to their own education, work ethic, technical ability, self-initiative, communication skills, academic skills, leadership skills, team player, and overall compared to other applicants. Applicants are ranked as Top 5%, Top 10%, Top 25%, Top 50%, or Below 50%. We converted these rankings to a score from 1 to 5 with 1 being Top 5% and 5 being Below 50%. We compared male and female applicants, the self-selected races of applicants, MD and DO applicants, and US vs IMG applicants using Kruskal-Wallis tests.

Results/Outcomes: In total, 356 standardized LOR were analyzed including 188 for male applicants and 168 for female applicants. Across all 11 domains, over 70% of applicants were ranked as “Top 5%” aside from technical ability and academics, where over 50% of applicants were ranked as “Top 5%.” Female applicants had significantly worse scores in professionalism, timelines, commitment to education, work ethic, and self-initiative (p-value < 0.05 for all, Table 1). Applicants who self-identified as black had significant worse scores in timeliness and academic skill (p-value < 0.05). DO applicants had significantly higher scores in commitment to education, academic skills, and overall compared to other applicants (p-value < 0.05 for all). IMG applicants had significantly lower scores for technical ability, leadership skills, and overall compared to other academics (p-value < 0.05 for all).

Conclusion/Discussion: This study highlights multiple issues with the current standardized LOR system for colorectal surgery. There is clear score inflation with over 70% of applicants being rated as “Top 5%” across most domains. This unfortunately makes the scoring system mostly useless and puts applicants who may be rated accurately at a tremendous disadvantage. This study also unfortunately likely uncovered bias with female applicants, black applicants, and IMG applicants having worse scores across multiple domains. The current standardized LOR system has unfortunately not helped combat performance inflation or bias. There are currently plans to change the LOR system, and based on this study, these changes are welcome and needed.