4.7 Article

Self-supervised temporal autoencoder for egocentric action segmentation

Journal

Publisher

PERGAMON-ELSEVIER SCIENCE LTD
DOI: 10.1016/j.engappai.2023.107092

Keywords

First-person vision systems; Temporal segmentation; Representation learning; Self-supervised learning

Ask authors/readers for more resources

In this study, a temporal-aware clustering method is proposed for egocentric action temporal segmentation. A self-supervised temporal autoencoder (SSTAE) is used to address the challenge of camera movement and abrupt visual content changes. The proposed approach outperforms the state-of-the-art methods in experimental evaluations.
Given an egocentric video, action temporal segmentation aims to temporally segment the video into basic units, each depicting an action. As the camera is constantly moving, some important objects may disappear in some consecutive frames and cause an abrupt change in the visual content. Recently works fail to deal with this condition in the absence of manually annotating abundant frames. In this study, we propose a temporal-aware clustering method for egocentric action temporal segmentation: a self-supervised temporal autoencoder (SSTAE). Instead of directly learning visual features, the SSTAE is implemented by encoding the preceding target frame and predicting subsequent frames in the temporal relationship domain, which takes into account the local temporal consistency. Our proposed algorithm is guided by the reconstruction and predicted losses. Consequently, local temporal contexts are naturally integrated into the feature representation, and a clustering step is performed. Experiments on three egocentric datasets demonstrate the our proposed approach outperforms the state-of-the-art methods by clustering Accuracy(ACC) 7.57%, Normalized Mutual Information(NMI) 8.17%, Adjusted Rand Index(ARI) 8.6%.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.7
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available