Multi-Level Fusion Net for hand pose estimation in hand-object interaction

This work is about solving a challenging problem of estimating the full 3D hand pose when a hand interacts with an unknown object. Compared to isolated single hand pose estimation, occlusion and interference induced by the manipulated object and the clutter background bring more difficulties for this task. Our proposed Multi-Level Fusion Net focuses on extracting more effective features to overcome these disadvantages by multi-level fusion design with a new end-to-end Convolutional Neural Network (CNN) framework. It takes cropped RGBD data from a single RGBD camera at free viewpoint as input without requiring additional handobject pre-segmentation and object or hand pre-modeling. Through extensive evaluations on public hand-object interaction dataset, we demonstrate the state-of-the-art performance of our method.

Automated summary

This work addresses the challenging problem of estimating the full 3D hand pose when a hand interacts with an unknown object. By focusing on extracting more effective features through multi-level fusion design, the proposed method demonstrates state-of-the-art performance on a public hand-object interaction dataset.