An Efficient Cross-Modality Self-Calibrated Network for Hyperspectral and Multispectral Image Fusion

Recently, deep convolutional neural network (CNN)-based hyperspectral and multispectral image fusion methods have shown significant performance. Nevertheless, the rich spatial and spectral details of hyperspectral images (HSIs) have not been fully explored, leaving room for further improve the representation ability of the model. In this article, we propose an efficient cross-modality self-calibrated network (CMSCN) for hyperspectral and multispectral image fusion. Specifically, we use a cross-modality nonlocal module (NL) to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) to get an enhanced LR-HSI. In addition, a novel cross-scale self-calibrated convolution structure is proposed to explore and exploit multiscale and hierarchical spatial-spectral features, which can improve the learning ability of the model. The introduced efficient spatial-spectral attention mechanism can calibrate the feature representation at different dimensions, thereby providing more efficient and accurate information for HSI reconstruction. Extensive experimental results on various HSIs demonstrate the superiority of our method in comparison with the state-of-the-art image fusion methods.

Automated summary

In this article, an efficient cross-modality self-calibrated network (CMSCN) is proposed for hyperspectral and multispectral image fusion. By combining a cross-modality nonlocal module and a cross-scale self-calibrated convolution structure, the learning ability of the model is improved. The introduced efficient spatial-spectral attention mechanism provides more accurate information for hyperspectral image reconstruction. Experimental results demonstrate the superiority of the proposed method over other image fusion methods.