Deep Fusion of Brain Structure-Function in Mild Cognitive Impairment

Multimodal fusion of different types of neural image data provides an irreplaceable opportunity to take advantages of complementary cross-modal information that may only partially be contained in single modality. To jointly analyze multimodal data, deep neural networks can be especially useful because many studies have suggested that deep learning strategy is very efficient to reveal complex and non-linear relations buried in the data. However, most deep models, e.g., convolutional neural network and its numerous extensions, can only operate on regular Euclidean data like voxels in 3D MRI. The inter-related and hidden structures that beyond the grid neighbors, such as brain connectivity, may be over-looked. Moreover, how to effectively incorporate neuroscience knowledge into multimodal data fusion with a single deep framework is understudied. In this work, we developed a graph-based deep neural network to simultaneously model brain structure and function in Mild Cognitive Impairment (MCI): the topology of the graph is initialized using structural network (from diffusion MRI) and iteratively updated by incorporating functional information (from functional MRI) to maximize the capability of differenti-ating MCI patients from elderly normal controls. This resulted in a new connectome by exploring deep relations between brain structure and function in MCI patients and we named it as Deep Brain Connec-tome. Though deep brain connectome is learned individually, it shows consistent patterns of alteration comparing to structural network at group level. With deep brain connectome, our developed deep model can achieve 92.7% classification accuracy on ADNI dataset. (c) 2021 Elsevier B.V. All rights reserved.

Automated summary

Multimodal fusion of neural image data offers a unique opportunity to utilize cross-modal information using deep neural networks, with a focus on revealing complex relations buried in the data. In this study, a graph-based deep neural network was developed to model brain structure and function in Mild Cognitive Impairment patients, resulting in a new connectome named Deep Brain Connectome. The deep model showed consistent alteration patterns compared to structural network at the group level and achieved high classification accuracy on ADNI dataset.