Fast and Unsupervised Action Boundary Detection for Action Segmentation

To deal with the great number of untrimmed videos produced every day, we propose an efficient unsupervised action segmentation method by detecting boundaries, named action boundary detection (ABD). In particular, the proposed method has the following advantages: no training stage and low-latency inference. To detect action boundaries, we estimate the similarities across smoothed frames, which inherently have the properties of internal consistency within actions and external discrepancy across actions. Under this circumstance, we successfully transfer the boundary detection task into the change point detection based on the similarity. Then, non-maximum suppression (NMS) is conducted in local windows to select the smallest points as candidate boundaries. In addition, a clustering algorithm is followed to refine the initial proposals. Moreover, we also extend ABD to the online setting, which enables real-time action segmentation in long untrimmed videos. By evaluating on four challenging datasets, our method achieves state-of-the-art performance. Moreover, thanks to the efficiency of ABD, we achieve the best trade-off between the accuracy and the inference time compared with existing unsupervised approaches.

Automated summary

This article proposes an efficient unsupervised action segmentation method by detecting boundaries, named action boundary detection (ABD), to handle the large number of untrimmed videos produced daily. The method has the advantages of no training stage and low-latency inference. By estimating similarities across smoothed frames, the boundary detection task is successfully transformed into change point detection based on similarity. Non-maximum suppression and a clustering algorithm are used to refine the initial proposals. The method achieves state-of-the-art performance and the best trade-off between accuracy and inference time compared to existing unsupervised approaches.