Deep Learning-Based Annotation Transfer between Molecular Imaging Modalities: An Automated Workflow for Multimodal Data Integration

An ever-increasing array of imaging technologies are being used in the study of complex biological samples, each of which provides complementary, occasionally overlapping information at different length scales and spatial resolutions. It is important to understand the information provided by one technique in the context of the other to achieve a more holistic overview of such complex samples. One way to achieve this is to use annotations from one modality to investigate additional modalities. For microscopy-based techniques, these annotations could be manually generated using digital pathology software or automatically generated by machine learning (including deep learning) methods. Here, we present a generic method for using annotations from one microscopy modality to extract information from complementary modalities. We also present a fast, general, multimodal registration workflow [evaluated on multiple mass spectrometry imaging (MSI) modalities, matrix-assisted laser desorption/ionization, desorption electrospray ionization, and rapid evaporative ionization mass spectrometry] for automatic alignment of complex data sets, demonstrating an order of magnitude speed-up compared to previously published work. To demonstrate the power of the annotation transfer and multimodal registration workflows, we combine MSI, histological staining (such as hematoxylin and eosin), and deep learning (automatic annotation of histology images) to investigate a pancreatic cancer mouse model. Neoplastic pancreatic tissue regions, which were histologically indistinguishable from one another, were observed to be metabolically different. We demonstrate the use of the proposed methods to better understand tumor heterogeneity and the tumor microenvironment by transferring machine learning results freely between the two modalities.

Automated summary

The article introduces a generic method for extracting information from different modalities using annotations from one microscopy modality and presents a fast, general multimodal registration workflow for automatic alignment of complex data sets. Experimental results demonstrate that the proposed methods help better understand tumor heterogeneity and the tumor microenvironment.