Joint Hand-Object 3D Reconstruction From a Single Image With Cross-Branch Feature Fusion

Accurate 3D reconstruction of the hand and object shape from a hand-object image is important for understanding human-object interaction as well as human daily activities. Different from bare hand pose estimation, hand-object interaction poses a strong constraint on both the hand and its manipulated object, which suggests that hand configuration may be crucial contextual information for the object, and vice versa. However, current approaches address this task by training a two-branch network to reconstruct the hand and object separately with little communication between the two branches. In this work, we propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We extensively investigate cross-branch feature fusion architectures with MLP or LSTM units. Among the investigated architectures, a variant with LSTM units that enhances object feature with hand feature shows the best performance gain. Moreover, we employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map, which further improves the reconstruction accuracy. Experiments conducted on public datasets demonstrate that our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.

Automated summary

Accurately reconstructing 3D hand and object shapes is crucial for understanding human-object interaction. Unlike traditional bare hand pose estimation, the interaction between hand and object imposes constraints on both, suggesting that considering hand configuration as contextual information for the object is important. Current approaches often use separate branches to reconstruct the hand and object, with little communication between them. This study proposes a joint approach in feature space, exploring the reciprocity between hand and object branches. Through cross-branch feature fusion architectures and an auxiliary depth estimation module, the proposed method outperforms existing approaches in terms of reconstruction accuracy.