NT1 - AI-Powered Semantic Segmentation Model for Enhanced Ureteral Mapping and Real-Time Instrument Feedback in Robotic Colorectal Surgery

Purpose/Background: Accurate identification of the ureters during robotic colorectal surgery is essential to avoid intraoperative injury. The current methods for ureter detection, including dissection with direct visualization or ureteral stenting with or without indocyanine green (ICG) installation, can be time consuming and have several limitations. ICG visualization necessitates a dark, near-infrared operative field (Firefly mode), temporarily interrupting the procedure; and its reliability can be compromised in cases of dense adhesions, prior radiation, or significant intra-abdominal adiposity. Furthermore, there is currently no method for intraoperative live feedback to provide a warning during instrument and ureteral proximity.
This study aims to develop and validate a deep learning-based object detection model to enable ureter mapping without ICG and provide real-time instrument feedback for ureter proximity during dissection in robotic colorectal surgery. The ultimate goal is to use this model as a real-time application on the da Vinci robot to enable an additional method for intraoperative ureter detection during robotic colorectal surgery.

Methods/Interventions: Deidentified videos of robotic-assisted Low Anterior Resection (LAR) and sigmoid colectomies performed by colorectal surgeons were annotated for ureter location. A convolutional neural network (CNN) model was trained on these video segments to consistently recognize ureters among any video, independent of Firefly mode, and across different camera viewpoints in multiple operative fields. The model was programmed to automatically generate identification boxes over the ureters and instrument tips during each frame of live videos. This allowed for consistent ureter detection during camera movements and in various operative fields. The model was then applied to new, unseen robotic surgery videos where it automatically generated ureter and instrument detection boxes in real-time and issued a live warning signal during moments where the ureter and instrument tips were in close proximity. Figure I shows example pictures captured during the live video which demonstrate the model’s real-time interface and overlay of detection boxes and warning signals.

Results/Outcomes: The trained model demonstrated high accuracy with mean Average Position score (mAP) of 0.932 which is considerably better than most real-world models (mAP range: 0.5-0.9). The model also showed a high F1 score of 0.883 demonstrating balance between precision and recall. However, the model misclassified 14.9% of evaluated sample frames as background, most commonly during visual fields with low contrast or complex anatomical conditions. Model enhancements are being implemented to improve performance in these difficult conditions.

Conclusion/Discussion: Our AI-powered semantic segmentation model enables consistent ureteral visualization without ICG and provides real-time feedback for ureter-instrument proximity during robotic colorectal surgery. The object detection model automatically generates detection boxes for the ureter and instrument tips in real-time when applied to new, unseen robotic surgery videos regardless of camera movement or varying operative fields. We are in the process of model refinements including minimizing the overlay detection box width, establishing a segmented mapping algorithm for visualization of the entire ureter course intraoperatively, and developing a detection model for three-dimensional ureter depth.

Detection Model's Real-Time Interface with Example Pictures Captured from Live Robotic Surgery Videos
Figure I: Row 1 shows operative field during IMA dissection. Row 2 shows operative field during rectal dissection. Column 1 shows original video frame. Column 2 shows AI-automated object identification for the ureters (green box) and instrument tips (blue box) taken in real-time during live video. Column 3 shows the warning signals activated during proximity of instrument tips to the ureter with and without Firefly mode.