Weakly Supervised Action Selection Learning in Video

Localizing actions in video is a core task in computer vision. The weakly supervised temporal localization problem investigates whether this task can be adequately solved with only video-level labels, significantly reducing the amount of expensive and error-prone annotation that is required. A common approach is to train a frame-level classifier where frames with the highest class probability are selected to make a video-level prediction. Frame-level activations are then used for localization. However, the absence of frame-level annotations cause the classifier to impart class bias on every frame. To address this, we propose the Action Selection Learning (ASL) approach to capture the general concept of action, a property we refer to as actionness. Under ASL, the model is trained with a novel class-agnostic task to predict which frames will be selected by the classifier. Empirically, we show that ASL outperforms leading baselines on two popular benchmarks THUMOS-14 and ActivityNet-I.2, with 10.3% and 5.7% relative improvement respectively. We further anab=e the properties of ASL and demonstrate the importance of actionness. Full code for this work is available here: https://github.com/layer6ai labs/ASL.

Automated summary

The paper introduces the Action Selection Learning (ASL) approach to address the action localization problem in videos. By training the model to predict which frames will be selected by the classifier, it effectively captures the concept of actions. Empirically, ASL outperforms leading baselines on two popular benchmarks.