LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animals

LiftPose3D infers three-dimensional poses from two-dimensional data or from limited three-dimensional data. The approach is illustrated for videos of behaving Drosophila, mice, rats and macaques. Markerless three-dimensional (3D) pose estimation has become an indispensable tool for kinematic studies of laboratory animals. Most current methods recover 3D poses by multi-view triangulation of deep network-based two-dimensional (2D) pose estimates. However, triangulation requires multiple synchronized cameras and elaborate calibration protocols that hinder its widespread adoption in laboratory studies. Here we describe LiftPose3D, a deep network-based method that overcomes these barriers by reconstructing 3D poses from a single 2D camera view. We illustrate LiftPose3D's versatility by applying it to multiple experimental systems using flies, mice, rats and macaques, and in circumstances where 3D triangulation is impractical or impossible. Our framework achieves accurate lifting for stereotypical and nonstereotypical behaviors from different camera angles. Thus, LiftPose3D permits high-quality 3D pose estimation in the absence of complex camera arrays and tedious calibration procedures and despite occluded body parts in freely behaving animals.

Automated summary

LiftPose3D is a deep network-based method that reconstructs 3D poses from a single 2D camera view, overcoming the limitations of traditional triangulation methods. It has been demonstrated to be versatile and effective in analyzing behaviors of various experimental animals, even in scenarios where 3D triangulation is not feasible. The framework enables accurate 3D pose estimation without the need for complex camera arrays and tedious calibration procedures, even in the presence of occluded body parts in freely behaving animals.