PVLI: potentially visible layered image for real-time ray tracing

Novel view synthesis is frequently employed in video streaming, temporal upsampling, or virtual reality. We propose a new representation, potentially visible layered image (PVLI), that uses a combination of a potentially visible set of the scene geometry and layered color images. PVLI encodes the depth implicitly and enables cheap run-time reconstruction. Furthermore, PVLI can also be used to reconstruct pixel and layer connectivities, which is crucial for filtering and post-processing of the rendered images. We use PVLIs to achieve local and server-based real-time ray tracing. In the first case, PVLIs are used as a basis for temporal and spatial upsampling of ray-traced illumination. In the second case, PVLIs are compressed, streamed over the network, and then used by a thin client to perform temporal and spatial upsampling and to hide latency. To shade the view, we use path tracing, accounting for effects such as soft shadows, global illumination, and physically based refraction. Our method supports dynamic lighting, and up to a limited extent, it also handles view-dependent surface interactions.

Automated summary

Novel view synthesis is improved by using a new representation called potentially visible layered image (PVLI) that encodes the depth implicitly and allows for cheap run-time reconstruction. PVLI also enables reconstruction of pixel and layer connectivities, which is important for filtering and post-processing of rendered images. The method supports real-time ray tracing for temporal and spatial upsampling of ray-traced illumination, as well as network streaming and thin client utilization for latency hiding and upsampling.