期刊
ACM TRANSACTIONS ON GRAPHICS
卷 41, 期 6, 页码 -出版社
ASSOC COMPUTING MACHINERY
DOI: 10.1145/3550454.3555451
关键词
virtual human; neural rendering; human modeling; human performance capture
资金
- Shanghai YangFan Program [21YF1429500]
- Shanghai local college capacity building program [22010502800]
- NSFC [61976138, 61977047]
- National Key Research and Development Program [2018YFB2100 500]
- STCSM [2015F0203000-06]
- SOME [2019-01-07-00-01-E00003]
In this paper, a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos is presented. The approach combines traditional animated mesh workflow with efficient neural techniques to generate high-quality surfaces and animated meshes. Various rendering schemes are discussed to achieve flexible rendering under different bandwidth settings.
We have recently seen tremendous progress in the neural advances for photo-real human modeling and rendering. However, it's still challenging to integrate them into an existing mesh-based pipeline for downstream applications. In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos. Our core intuition is to bridge the traditional animated mesh workflow with a new class of highly efficient neural techniques. We first introduce a neural surface reconstructor for high-quality surface generation in minutes. It marries the implicit volumetric rendering of the truncated signed distance field (TSDF) with multi-resolution hash encoding. We further propose a hybrid neural tracker to generate animated meshes, which combines explicit non-rigid tracking with implicit dynamic deformation in a self-supervised framework. The former provides the coarse warping back into the canonical space, while the latter implicit one further predicts the displacements using the 4D hash encoding as in our reconstructor. Then, we discuss the rendering schemes using the obtained animated meshes, ranging from dynamic texturing to lumigraph rendering under various bandwidth settings. To strike an intricate balance between quality and bandwidth, we propose a hierarchical solution by first rendering 6 virtual views covering the performer and then conducting occlusion-aware neural texture blending. We demonstrate the efficacy of our approach in a variety of mesh-based applications and photo-realistic free-view experiences on various platforms, i.e., inserting virtual human performances into real environments through mobile AR or immersively watching talent shows with VR headsets.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据