Real-Time Decoding of Snapshot Compressive Imaging Using Tensor FISTA-Net

Snapshot compressive imaging (SCI) cameras compress high-speed videos or hyperspectral images into measurement frames. However, decoding the data frames from measurement frames is compute-intensive. Existing state-of-the-art decoding algorithms suffer from low decoding quality or heavy running time or both, which are not practical for real-time applications. In this article, we exploit the powerful learning ability of deep neural networks (DNN) and propose a novel tensor fast iterative shrinkage-thresholding algorithm net (Tensor FISTA-Net) as a real-time decoder for SCI cameras. Since SCI cameras have an accurate physical model, we can trade training time for the decoding time by generating abundant synthetic data and training a decoder on the cloud. Tensor FISTA-Net not only learns a sparse representation of the frames through convolution layers but also reduces the decoding time and memory consumption significantly through tensor operations, which makes Tensor FISTA-Net an appropriate approach for a real-time decoder. Our proposed Tensor FISTA-Net obtains an average PSNR improvement of 0.79-2.84 dB (video images) and 2.61-4.43 dB (hyperspectral images) over the state-of-the-art algorithms, along with more clear and detailed visual results on real SCI datasets, Hammer and Wheel, respectively. Our Tensor FISTA-Net reaches 45 frames per second in video datasets and 70 frames per second in hyperspectral datasets, meeting the real-time requirement. Besides, the trained model occupies only a 12-MB memory footprint, making it applicable to real-time Internet of Things (IoT) applications.

Automated summary

This article proposes a novel tensor fast iterative shrinkage-thresholding algorithm net (Tensor FISTA-Net) as a real-time decoder for snapshot compressive imaging (SCI) cameras, utilizing the powerful learning ability of deep neural networks (DNN). It learns a sparse representation of the frames through convolution layers and significantly reduces decoding time and memory consumption through tensor operations. The experimental results show that Tensor FISTA-Net achieves an average PSNR improvement of 0.79-2.84 dB (video images) and 2.61-4.43 dB (hyperspectral images) over the state-of-the-art algorithms, with clearer and more detailed visual results on real SCI datasets. It reaches 45 frames per second in video datasets and 70 frames per second in hyperspectral datasets, meeting the real-time requirement, occupying only a 12-MB memory footprint.