NHK Meta Studio: A Compact Volumetric TV Studio for 3-D Reconstruction

Many studies have been conducted on the 3D reconstruction of subjects using multiple fixed cameras. Accepting the trade-off between the number of cameras and reconstruction quality, our studio is designed to capture high-quality models of one or two subjects for TV program use. Several cameras are mounted on a hemispherical dome with the stage in the center and a cloth cover on the frame for chroma-keying. The optimal camera numbers and placements for reconstruction were determined by simulation, and the 3D reconstruction was performed as a point cloud by a combination of visual hull and stereo matching. The quality was still not high enough, however, so we also added a surface light field to the point cloud to obtain the weighted average of rays from camera images close to the viewpoint. In the final stage, the images were then combined to the video, and errors generated during the reconstruction were compensated for by use of a deep neural network (DNN) for video translation. An offline processing studio has been built as a preliminary step towards real-time processing, and the reconstructed 3D images have been evaluated subjectively for a number of subjects. These studies confirm the effectiveness of this studio design.

Automated summary

This study conducted 3D reconstruction experiments using multiple fixed cameras. To improve the image quality, the optimal number and placement of cameras were determined and 3D reconstruction was performed using visual hull and stereo matching. Additionally, the addition of a surface light field and the use of a deep neural network for video translation further enhanced the reconstruction quality.