期刊
PATTERN RECOGNITION
卷 104, 期 -, 页码 -出版社
ELSEVIER SCI LTD
DOI: 10.1016/j.patcog.2020.107356
关键词
Action recognition; multi-modality; skeleton-guided
资金
- National Natural Science Foundation of China [61836008, 61632019]
Single-modality human action recognition on RGB or skeleton has been extensively studied. Each of these two modalities has its own advantages as well as limitations, because they depict action from different perspectives. The feature of different modalities can complement each other for describing actions. Therefore, it is meaningful to fuse these two modalities using their complementarity for action recognition. However, existing multimodal methods fail to fully exploit the complementarity of RGB and skeleton modalities. In this paper, we propose a Skeleton-Guided Multimodal Network (SGM-Net) for human action recognition. The proposed method takes full use of the complementarity of these two modalities at semantic feature level. From the technical perspective, we introduce a guided block, the key component of SGM-Net. It enables skeleton feature to guide on RGB feature, so that the important RGB information strongly related to the action is enhanced. Moreover, in the guided block, two schemes of correlation operation are explored. We perform a series of ablation experiments to verify the effectiveness of the guided block. The experimental results show that our approach achieves state-of-the-art performance over the existing methods on NTU and Sub-JHMDB datasets. (C) 2020 Elsevier Ltd. All rights reserved.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据