Geometric Deep Neural Network Using Rigid and Non-Rigid Transformations for Landmark-Based Human Behavior Analysis

Deep learning architectures, albeit successful in most computer vision tasks, were designed for data with an underlying Euclidean structure, which is not usually fulfilled since pre-processed data may lie on a non-linear space. In this article, we propose a geometric deep learning approach using rigid and non-rigid transformations, named KShapenet, for 2D and 3D landmark-based human motion analysis. Landmark configuration sequences are first modeled as trajectories on Kendall's shape space and then mapped to a linear tangent space. The resulting structured data are then input to a deep learning architecture, which includes a layer that optimizes over rigid and non-rigid transformations of landmark configurations, followed by a CNN-LSTM network. We apply KShapenet to 3D human landmark sequences for action and gait recognition, and 2D facial landmark sequences for expression recognition, and demonstrate the competitiveness of the proposed approach with respect to state-of-the-art.

Automated summary

In this article, a geometric deep learning approach called KShapenet is proposed for 2D and 3D landmark-based human motion analysis, using both rigid and non-rigid transformations. The method demonstrates its competitiveness in action recognition, gait recognition, and expression recognition tasks with respect to the state-of-the-art techniques.