Artificial intelligence for multimodal data integration in oncology

In oncology, the patient state is characterized by a whole spectrum of modalities, ranging from radiology, his-tology, and genomics to electronic health records. Current artificial intelligence (AI) models operate mainly in the realm of a single modality, neglecting the broader clinical context, which inevitably diminishes their po-tential. Integration of different data modalities provides opportunities to increase robustness and accuracy of diagnostic and prognostic models, bringing AI closer to clinical practice. AI models are also capable of discovering novel patterns within and across modalities suitable for explaining differences in patient out-comes or treatment resistance. The insights gleaned from such models can guide exploration studies and contribute to the discovery of novel biomarkers and therapeutic targets. To support these advances, here we present a synopsis of AI methods and strategies for multimodal data fusion and association discovery. We outline approaches for AI interpretability and directions for AI-driven exploration through multimodal data interconnections. We examine challenges in clinical adoption and discuss emerging solutions.

Automated summary

In oncology, the integration of multimodal data in artificial intelligence (AI) models is crucial for improving diagnostic and prognostic accuracy. Current AI models primarily operate within a single modality, neglecting the broader clinical context. By incorporating different data modalities, AI models can enhance their robustness and accuracy, bringing them closer to clinical practice.