Neural Radiance Flow for 4D View Synthesis and Video Processing

We present a method, Neural Radiance Flow (NeRFlow), to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images. Key to our approach is the use of a neural implicit representation that learns to capture the 3D occupancy, radiance, and dynamics of the scene. By enforcing consistency across different modalities, our representation enables multi-view rendering in diverse dynamic scenes, including water pouring, robotic interaction, and real images, outperforming state-of-the-art methods for spatial-temporal view synthesis. Our approach works even when being provided only a single monocular real video. We further demonstrate that the learned representation can serve as an implicit scene prior, enabling video processing tasks such as image super-resolution and de-noising without any additional supervision.

Automated summary

The method utilizes Neural Radiance Flow (NeRFlow) to learn a 4D spatial-temporal representation of dynamic scenes from RGB images. By using a neural implicit representation, it captures 3D occupancy, radiance, and dynamics of scenes, enabling multi-view rendering and video processing tasks without additional supervision.