Focalized contrastive view-invariant learning for self-supervised skeleton-based action recognition

Learning view-invariant representation is a key to improving feature discrimination power for skeleton -based action recognition. Existing approaches cannot effectively remove the impact of viewpoint due to the implicit view-dependent representations. In this work, we propose a self-supervised framework called Focalized Contrastive View-invariant Learning (FoCoViL), which significantly suppresses the view-specific information on the representation space where the viewpoints are coarsely aligned. By maximizing mutual information with an effective contrastive loss between multi-view sample pairs, FoCoViL associates actions with common view-invariant properties and simultaneously separates the dissimilar ones. We further propose an adaptive focalization method based on pairwise similarity to enhance contrastive learning for a clearer cluster boundary in the learned space. Different from many existing self-supervised representation learning work that rely heavily on supervised classifiers, FoCoViL performs well on both unsupervised and supervised classifiers with superior recognition perfor-mance. Extensive experiments also show that the proposed contrastive-based focalization generates a more discriminative latent representation.(c) 2023 Elsevier B.V. All rights reserved.

Automated summary

In this work, a self-supervised framework called FoCoViL is proposed, which associates actions with common view-invariant properties and simultaneously separates dissimilar viewpoints by maximizing mutual information between multi-view sample pairs. An adaptive focalization method based on pairwise similarity is further proposed to enhance contrastive learning for a clearer cluster boundary. FoCoViL performs well on both unsupervised and supervised classifiers, and the proposed contrastive-based focalization generates a more discriminative latent representation.