PSRT: Pyramid Shuffle-and-Reshuffle Transformer for Multispectral and Hyperspectral Image Fusion

A Transformer has received a lot of attention in computer vision. Because of global self-attention, the computational complexity of Transformer is quadratic with the number of tokens, leading to limitations for practical applications. Hence, the computational complexity issue can be efficiently resolved by computing the self-attention in groups of smaller fixed-size windows. In this article, we propose a novel pyramid Shuffleand-Reshuffle Transformer (PSRT) for the task of multispectral and hyperspectral image fusion (MHIF). Considering the strong correlation among different patches in remote sensing images and complementary information among patches with high similarity, we design Shuffle-and-Reshuffle (SaR) modules to consider the information interaction among global patches in an efficient manner. Besides, using pyramid structures based on window self-attention, the detail extraction is supported. Extensive experiments on four widely used benchmark datasets demonstrate the superiority of the proposed PSRT with a few parameters compared with several state-of-the-art approaches. The related code is available at https://github.com/Dengshangqi/PSRThttps://github.com/Deng-shangqi/PSRT.

Automated summary

In this article, a novel pyramid Shuffleand-Reshuffle Transformer (PSRT) algorithm is proposed for multispectral and hyperspectral image fusion (MHIF). By designing Shuffle-and-Reshuffle (SaR) modules and using pyramid structures based on window self-attention, PSRT efficiently resolves the computational complexity issue of Transformer. Experimental results demonstrate the superiority of PSRT compared to several state-of-the-art approaches on multiple benchmark datasets.