EP670 - Assessing Endoscopic Response to Preoperative Chemoradiotherapy in Locally Advanced Rectal Cancer using Siamese Multi-task Deep Learning

Purpose/Background: With the emergence of organ preservation strategies, accurate evaluation of treatment response plays a pivotal role in making treatment decisions for each patient. To improve the accuracy of predicting the effect of preoperative chemoradiotherapy (PCRT), we developed a Siamese multi-task deep learning of convolutional neural networks (CNNs) to interpret endoscopic images and assess the response to preoperative treatments.

Methods/Interventions: Endoscopic images from patients with locally advanced rectal cancer (LARC) (cT3 or higher, clinically node-positive) who underwent surgery after preoperative chemoradiotherapy between 2018 and 2023 at a single institution were retrospectively reviewed. Patients were classified into response and no response groups based on pathologic reports. A RectAi Siamese multi-task network using the pre- and post-treatment images for classification, segmentation and reconstruction simultaneously was employed to accurately assess the response to preoperative chemoradiotherapy. A two-stage training process was utilized: first, the EfficientNet-B7 encoder was pre-trained on a dataset of 199 images labeled and verified by clinical experts, and then the Siamese network, incorporating this pre-trained encoder, was fine-tuned on a small but carefully curated dataset comprising 85 patients (53 for training, 8 for validation, and 24 for testing). Extensive data augmentation techniques were applied to address the challenges posed by the limited dataset size.

Results/Outcomes: A total of 284 images from 85 patients were included in the study. The average age of the patients was 58.7 years, with 58 (68.2%) male and 27 (31.7%) female. Endoscopy was first performed at diagnosis before PCRT and then 4 weeks after PCRT for response evaluation. Patients with total or near-total response were grouped as the response group, while patients with moderate to minimal response were grouped as the no response group, according to the tumor regression grade in the pathology report after surgery. 45 patients (52.9%) had no response to PCRT, while 40 patients (47.1%) showed a response, with 19 patients (22.3%) showing complete response to PCRT.
The model showed poor prediction when the endoscopic image contained extraneous elements, such as remaining feces or blood around the mass. Additionally, if the tumor size was markedly reduced but had remaining tumor cells, or when the lesion was extensive and most of the remaining tissue was large but fibrotic, the prediction was also less accurate. By combining tasks and training the model in a multi-task learning framework, EfficientNet-B7 with all tasks demonstrated the highest diagnostic performance. The area under the curve (AUC) was 75.0%, with a Dice score of 0.9104 and a classification F1 score of 0.7272.

Conclusion/Discussion: This pilot study presents a novel method for assessing endoscopic treatment response to preoperative chemoradiotherapy, using a Siamese multi-task deep learning network with an EfficientNet-B7 backbone pre-trained on pseudo-labeled data. This approach demonstrates significant potential in supporting clinical decision-making for patients with LARC undergoing various neoadjuvant treatments.

Figure 1. Performance Evaluation and Model Processes in RectAI: Multitask Learning, Classification Accuracy, and Segmentation Outcomes.
A) Each model process in RectAI: Combination of tasks and training of the model within a multitask learning framework. B) Confusion matrices for different models trained under various task combinations in the RectAI framework. Each matrix shows the distribution of correctly and incorrectly classified results for Class 0 (no response) and Class 1 (response). C) Examples of model predictions, demonstrating both correct and incorrect cases. The figure displays two sets of input images, along with their corresponding reconstructions, segmentation labels, and segmentation predictions. The top row shows a case where the model correctly classifies and segments the lesion, while the bottom row highlights a case where the model fails to predict accurately.

Table 1. Results of the RectAI Model on Various Multitask Combinations with Different Backbones
MSE, Mean Squared Error; PSNR, Peak Signal-to-Noise Ratio; DSC, Dice Similarity Coefficient (DSC); IoU, Intersection over Union; Rec, Reconstruction; Cls, Classification; Seg, Segementation