MVFI-Net: Motion-Aware Video Frame Interpolation Network

Video frame interpolation (VFI) is to synthesize the intermediate frame given successive frames. Most existing learning-based VFI methods generate each target pixel by using the warping operation with either one predicted kernel or flow, or both. However, their performances are often degraded due to the issues on the limited direction and scope of the reference regions, especially encountering complex motions. In this paper, we propose a novel motion-aware VFI network (MVFI-Net) to address these issues. One of the key novelties of our method lies in the newly developed warping operation, i.e., motion-aware convolution (MAC). By predicting multiple extensible temporal motion vectors (MVs) and filter kernels for each target pixel, the direction and scope could be enlarged simultaneously. Besides, we first attempt to incorporate the pyramid structure into the kernel-based VFI, which can decompose large motions into smaller scales to improve the prediction efficiency. The quantitative and qualitative experimental results have demonstrated the proposed method delivers the state-of-the-art performance on the diverse benchmarks with various resolutions. Our codes are available at https://github.com/MediaLabVFI/MVFI-Net.

Automated summary

Video frame interpolation (VFI) aims to generate intermediate frames between successive frames. Existing learning-based methods often suffer from limited reference regions and complex motions. This paper proposes a novel motion-aware VFI network (MVFI-Net) which overcomes these issues by introducing a motion-aware convolution operation and incorporating pyramid structure. Experimental results demonstrate that the proposed method achieves state-of-the-art performance on diverse benchmarks.