Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

Many learning-based algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI). However, CNN-based methods show limitations in capturing long-range dependencies. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate multi-head self-attention (MSA) between some tokens that are unrelated in content. In this paper, we propose a novel Transformerbased method, coarse-to-fine sparse Transformer (CST), firstly embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing. Comprehensive experiments show that our CST significantly outperforms stateof-the-art methods while requiring cheaper computational costs. https://github.com/caiyuanhao1998/MST

Automated summary

This paper proposes a novel Transformer-based method that embeds sparsity in deep learning for hyperspectral imaging reconstruction. By using a coarse-to-fine strategy, the method utilizes a spectra-aware screening mechanism and a spectra-aggregation hashing multi-head self-attention to achieve high-quality results in HSI reconstruction.