Stereo Cross-Attention Network for Unregistered Hyperspectral and Multispectral Image Fusion

The necessary prerequisite for effective data fusion is the strict registration of low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs). However, registration requires a complex process that takes into account the effects of light, imaging angle, and geometric distortion of the image during acquisition. Therefore, to avoid complex registration, we focused on developing an unregistered HSI and MSI fusion method for pixel shifting, obtaining fused images with high resolution, high signal-to-noise ratio, and feature identifiability. We identified that the unregistered LR-HSI and HR-MSI in the case of pixel shift are very similar to the disparity maps in stereo vision. Inspired by this, we simulate the structure of stereo cameras to propose a stereo cross-attention network (SCANet) to achieve an accurate fusion of unregistered LR-HSI and HR-MSI. Considering the model complexity and computing efficiency, we design a simple and stackable stereo cross-fusion block (SCFBlock) based on a Transformer to simulate the process of light entering the left and right cameras by extracting the abstract features of the images. Moreover, the purpose of cross-convergence fusion self-attention (CCFSA) is to learn cross-complementary attention and collect contextual information in horizontal and vertical directions to fuse unregistered images using multidirectional cross-view information. We have conducted extensive experiments on Pavia University (PaviaU), Chikusei, and PYLake datasets. The results show that the SCANet achieves superior or competitive performance in fusing unregistered LR-HSI and HR-MSI in comparison with the other competitors.

Automated summary

This research proposes an efficient method for fusing unregistered low-resolution hyperspectral images and high-resolution multispectral images. By using pixel shifting, the method achieves high-resolution, high signal-to-noise ratio, and feature identifiability in the fused images. The research also introduces a simple and stackable fusion block and a stereo cross-attention network for accurate fusion.