NeX360: Real-Time All-Around View Synthesis With Neural Basis Expansion

We present NeX, a new approach to novel view synthesis based on enhancements of multiplane images (MPI) that can reproduce view-dependent effects in real time. Unlike traditional MPI, our technique parameterizes each pixel as a linear combination of spherical basis functions learned from a neural network to model view-dependent effects and uses a hybrid implicit-explicit modeling strategy to improve fine detail. Moreover, we also present an extension to NeX, which leverages knowledge distillation to train multiple MPIs for unbounded 360 degrees scenes. Our method is evaluated on several benchmark datasets: NeRF-Synthetic dataset, Light Field dataset, Real Forward-Facing dataset, Space dataset, as well as Shiny, our new dataset that contains significantly more challenging view-dependent effects, such as the rainbow reflections on the CD. Our method outperforms other real-time rendering approaches on PSNR, SSIM, and LPIPS and can render unbounded 360 degrees scenes in real time.

Automated summary

We propose NeX, a novel approach for real-time novel view synthesis based on enhanced multiplane images (MPI) that can reproduce view-dependent effects. Our technique uses spherical basis functions learned from a neural network to parameterize each pixel and improve fine detail through a hybrid implicit-explicit modeling strategy. Additionally, we introduce an extension to NeX that utilizes knowledge distillation to train multiple MPIs for unbounded 360-degree scenes. Evaluations on multiple benchmark datasets demonstrate that our method outperforms other real-time rendering approaches and can handle challenging view-dependent effects such as rainbow reflections on CDs.