F-DARTS: Foveated Differentiable Architecture Search Based Multimodal Medical Image Fusion

Multimodal medical image fusion (MMIF) is highly significant in such fields as disease diagnosis and treatment. The traditional MMIF methods are difficult to provide satisfactory fusion accuracy and robustness due to the influence of such possible human-crafted components as image transform and fusion strategies. Existing deep learning based fusion methods are generally difficult to ensure image fusion effect due to the adoption of a human-designed network structure and a relatively simple loss function and the ignorance of human visual characteristics during weight learning. To address these issues, we have presented the foveated differentiable architecture search (F-DARTS) based unsupervised MMIF method. In this method, the foveation operator is introduced into the weight learning process to fully explore human visual characteristics for the effective image fusion. Meanwhile, a distinctive unsupervised loss function is designed for network training by integrating mutual information, sum of the correlations of differences, structural similarity and edge preservation value. Based on the presented foveation operator and loss function, an end-to-end encoder-decoder network architecture will be searched using the F-DARTS to produce the fused image. Experimental results on three multimodal medical image datasets demonstrate that the F-DARTS performs better than several traditional and deep learning based fusion methods by providing visually superior fused results and better objective evaluation metrics.

Automated summary

Multimodal medical image fusion is highly significant in disease diagnosis and treatment. Traditional methods and deep learning methods have limitations. To address these issues, we propose an unsupervised multimodal medical image fusion method based on foveated differentiable architecture search (F-DARTS). The method introduces the foveation operator and a distinctive unsupervised loss function. Experimental results demonstrate that the proposed method outperforms traditional methods and deep learning methods in visual effect and objective evaluation.