4.8 Article

MannequinChallenge: Learning the Depths of Moving People by Watching Frozen People

Journal

Publisher

IEEE COMPUTER SOC
DOI: 10.1109/TPAMI.2020.2974454

Keywords

Cameras; Three-dimensional displays; Cleaning; Internet; Image reconstruction; Geometry; Training; Depth prediction; mannequin challenge; dynamic scene reconstruction

Ask authors/readers for more resources

The method leverages human depth priors and multi-view stereo reconstruction to generate training data, then uses motion parallax cues from static areas for depth prediction, showing improvement over state-of-the-art monocular depth prediction methods and demonstrating various 3D effects.
We present a method for predicting dense depth in scenarios where both a monocular camera and people in the scene are freely moving (right). Existing methods for recovering depth for dynamic, non-rigid objects from monocular video impose strong assumptions on the objects' motion and may only recover sparse depth. In this paper, we take a data-driven approach and learn human depth priors from a new source of data: thousands of Internet videos of people imitating mannequins, i.e., freezing in diverse, natural poses, while a hand-held camera tours the scene (left). Because people are stationary, geometric constraints hold, thus training data can be generated using multi-view stereo reconstruction. At inference time, our method uses motion parallax cues from the static areas of the scenes to guide the depth prediction. We evaluate our method on real-world sequences of complex human actions captured by a moving hand-held camera, show improvement over state-of-the-art monocular depth prediction methods, and demonstrate various 3D effects produced using our predicted depth.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.8
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available