期刊
ACM TRANSACTIONS ON GRAPHICS
卷 37, 期 6, 页码 -出版社
ASSOC COMPUTING MACHINERY
DOI: 10.1145/3272127.3275084
关键词
Image-based rendering; Free-viewpoint; Deep learning
资金
- NSF [CNS-1405847]
- NERC [NE/P019013/1]
- EU H2020 EMOTIVE project
- Rabin Ezra scholarship fund
- EPSRC [EP/K023578/1, EP/K02339X/1] Funding Source: UKRI
- NERC [NE/P019013/1] Funding Source: UKRI
Free-viewpoint image-based rendering (IBR) is a standing challenge. IBR methods combine warped versions of input photos to synthesize a novel view. The image quality of this combination is directly affected by geometric inaccuracies of multi-view stereo (MVS) reconstruction and by view-and image-dependent effects that produce artifacts when contributions from different input views are blended. We present a new deep learning approach to blending for IBR, in which we use held-out real image data to learn blending weights to combine input photo contributions. Our Deep Blending method requires us to address several challenges to achieve our goal of interactive free-viewpoint IBR navigation. We first need to provide sufficiently accurate geometry so the Convolutional Neural Network (CNN) can succeed in finding correct blending weights. We do this by combining two different MVS reconstructions with complementary accuracy vs. completeness tradeoffs. To tightly integrate learning in an interactive IBR system, we need to adapt our rendering algorithm to produce a fixed number of input layers that can then be blended by the CNN. We generate training data with a variety of captured scenes, using each input photo as ground truth in a held-out approach. We also design the network architecture and the training loss to provide high quality novel view synthesis, while reducing temporal flickering artifacts. Our results demonstrate free-viewpoint IBR in a wide variety of scenes, clearly surpassing previous methods in visual quality, especially when moving far from the input cameras.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据