Fast Online Video Super-Resolution with Deformable Attention Pyramid

Video super-resolution (VSR) has many applications that pose strict causal, real-time, and latency constraints, including video streaming and TV. We address the VSR problem under these settings, which poses additional important challenges since information from future frames is unavailable. Importantly, designing efficient, yet effective frame alignment and fusion modules remain central problems. In this work, we propose a recurrent VSR architecture based on a deformable attention pyramid (DAP). Our DAP aligns and integrates information from the recurrent state into the current frame prediction. To circumvent the computational cost of traditional attention-based methods, we only attend to a limited number of spatial locations, which are dynamically predicted by the DAP. Comprehensive experiments and analysis of the proposed key innovations show the effectiveness of our approach. We significantly reduce processing time and computational complexity in comparison to state-of-the-art methods, while maintaining a high performance. We surpass state-of-the-art method EDVR-M on two standard benchmarks with a speed-up of over 3x.

Automated summary

In this work, we propose a recurrent VSR architecture based on a deformable attention pyramid (DAP) to address the VSR problem with strict causal, real-time, and latency constraints. Our DAP aligns and integrates information from the recurrent state into the current frame prediction, overcoming the challenge of unavailable future frame information. By attending to a limited number of spatial locations dynamically predicted by the DAP, we reduce computational cost compared to traditional attention-based methods. Experimental results show the effectiveness of our approach, achieving a significant speed-up and surpassing state-of-the-art methods on benchmark tests.