A Tri-Attention fusion guided multi-modal segmentation network

In the field of multimodal segmentation, the correlation between different modalities can be considered for improving the segmentation results. Considering the correlation between different MR modalities, in this paper, we propose a multi-modality segmentation network guided by a novel tri-attention fusion. Our network includes N model-independent encoding paths with N image sources, a tri-attention fusion block, a dual-attention fusion block, and a decoding path. The model independent encoding paths can capture modality-specific features from the N modalities. Considering that not all the features extracted from the encoders are useful for segmentation, we propose to use dual attention based fusion to reweight the features along the modality and space paths, which can suppress less informative features and emphasize the useful ones for each modality at different positions. Since there exists a strong correlation between different modalities, based on the dual attention fusion block, we propose a correlation attention module to form the tri-attention fusion block. In the correlation attention module, a correlation description block is first used to learn the correlation between modalities and then a constraint based on the correlation is used to guide the network to learn the latent correlated features which are more relevant for segmentation. Finally, the obtained fused feature representation is projected by the decoder to obtain the segmentation results. Our experiment results tested on BraTS 2018 dataset for brain tumor segmentation demonstrate the effectiveness of our proposed method. (c) 2021 Elsevier Ltd. All rights reserved.

Automated summary

In this paper, a multi-modality segmentation network guided by a novel tri-attention fusion is proposed to improve the segmentation results by considering the correlation between different MR modalities. The network incorporates model-independent encoding paths, attention fusion blocks, and a decoding path to capture modality-specific features, reweight the features based on attention, and learn the latent correlated features using correlation attention. Experimental results on the BraTS 2018 dataset demonstrate the effectiveness of the proposed method.