Image-based Pose Representation for Action Recognition and Hand Gesture Recognition

In this paper, we propose an effective and compact image-based pose representation named Poseimage Pyramid, which encodes the spatial and temporal information of human pose or hand pose as an image pyramid. Poseimage is constructed by the normalized distance between pairwise joints, and it has the advantage of its invariant to similarity transformations. With our Poseimage representation we can design the pose based action recognition or hand gesture recognition model using existing image or video classification models. In order to adapt to different actions with a variety of movement speed, we design Poseimage Pyramid to encode the multi-scale temporal information of human pose or hand pose. Experiments demonstrate that our pose representation is effective, and we achieve state-of-the-art performance on the action recognition datasets and the hand gesture recognition datasets. Our pose presentation is also complementary to video and optical flow streams in the seminal action recognition network I3D, and we achieve the state-of-the-art performance on the JHMDB, HMDB and UCF101 datasets by integrating our pose representation with I3D.