Skeleton-Based Action Recognition with Directed Graph Neural Networks

The skeleton data have been widely used for the action recognition tasks since they can robustly accommodate dynamic circumstancesand complex backgrounds. In existing methods, both the joint and bone information in skeleton data have been proved to be of great help for action recognition tasks. However, how to incorporate these two types of data to best take advantage of the relationshipbetween joints and bones remains a problem to be solved. In this work, we represent the skeleton data as a directed acyclic graph (DAG) based on the kinematic dependency between the joints and bones in the naturalhuman body. A novel directedgraphneural network is designedspecially to extract the information ofjoints, bones and their relationshipsand make prediction based on the extractedfeatures. In addition, to betterfit the action recognitiontask, the topological structure of the graph is made adaptivebased on the training process, which brings notable improvement. Moreover, the motion information of the skeleton sequence is exploited and combined with the spatial information to further enhance the performance in a two-stream framework. Ourfinal model is tested on two large-scaledatasets, NTU-RGBD and Skeleton-Kinetics, and exceeds state-of-the-artperformance on both of them.