3.8 Proceedings Paper

Playable Environments: Video Manipulation in Space and Time

出版社

IEEE COMPUTER SOC
DOI: 10.1109/CVPR52688.2022.00357

关键词

-

资金

  1. ERC consolidator grant 4DReply [770784]
  2. EU H2020 project AI4Media [951911]

向作者/读者索取更多资源

We present Playable Environments, a novel representation for interactive video generation and manipulation. It allows the user to move objects in 3D and generate videos by providing a sequence of desired actions based on a single image at inference time. The actions are learned in an unsupervised manner and the camera can be controlled to achieve the desired viewpoint. Our method builds an environment state for each frame that can be manipulated using our proposed action module and rendered back to the image space with volumetric rendering. We also introduce two large-scale video datasets with significant camera movements to set a challenging benchmark. Playable environments enable creative applications such as 3D video generation, stylization, and manipulation that were not possible with prior video synthesis works.
We present Playable Environments-a new representation for interactive video generation and manipulation in space and time. With a single image at inference time, our novel framework allows the user to move objects in 3D while generating a video by providing a sequence of desired actions. The actions are learnt in an unsupervised manner. The camera can be controlled to get the desired viewpoint. Our method builds an environment state for each frame, which can be manipulated by our proposed action module and decoded back to the image space with volumetric rendering. To support diverse appearances of objects, we extend neural radiance fields with style-based modulation. Our method trains on a collection of various monocular videos requiring only the estimated camera parameters and 2D object locations. To set a challenging benchmark, we introduce two large scale video datasets with significant camera movements. As evidenced by our experiments, playable environments enable several creative applications not attainable by prior video synthesis works, including playable 3D video generation, stylization and manipulation(1).

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

3.8
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据