Predicting spatial esophageal changes in a multimodal longitudinal imaging study via a convolutional recurrent neural network

Acute esophagitis (AE) occurs among a significant number of patients with locally advanced lung cancer treated with radiotherapy. Early prediction of AE, indicated by esophageal wall expansion, is critical, as it can facilitate the redesign of treatment plans to reduce radiation-induced esophageal toxicity in an adaptive radiotherapy (ART) workflow. We have developed a novel machine learning framework to predict the patient-specific spatial presentation of the esophagus in the weeks following treatment, using magnetic resonance imaging (MRI)/ cone-beam CT (CBCT) scans acquired earlier in the 6 week radiotherapy course. Our algorithm captures the response patterns of the esophagus to radiation on a patch level, using a convolutional neural network. A recurrence neural network then parses the evolutionary patterns of the selected features in the time series, and produces a predicted esophagus-or-not label for each individual patch over future weeks. Finally, the esophagus is reconstructed, using all the predicted labels. The algorithm is trained and validated by means of similar to 250 000 patches taken from MRI scans acquired weekly from a variety of patients, and tested using both weekly MRI and CBCT scans under a leave-one-patient-out scheme. In addition, our approach is externally validated using a publicly available dataset (Hugo 2017). Using the first three weekly scans, the algorithm can predict the condition of the esophagus over the succeeding 3 weeks with a Dice coefficient of 0.83 +/- 0.04, estimate esophagus volume highly (0.98), correlated with the actual volume, using our institutional MRI/CBCT data. When evaluated using the external weekly CBCT data, the averaged Dice coefficient is 0.89 +/- 0.03. Our novel algorithm may prove useful in enabling radiation oncologists to monitor and detect AE in its early stages, and could potentially play an important role in the ART decision-making process.