Long-Term Action Dependence-Based Hierarchical Deep Association for Multi-Athlete Tracking in Sports Videos

Tracking multiple athletes in sports videos is a very challenging Multi-Object Tracking (MOT) task, as athletes generally share high similarity in appearance with large deformations. In this paper, unlike the existing hand-crafted solutions, we propose a novel and effective approach to this issue, which hierarchically associates detections of the same identity through discriminative and robust deep features. First, in detection association, we make use of athlete appearances and poses instead of traditional position cues to generate short tracklets for better initialization. Second, in tracklet association, a new deep architecture, namely Siamese Tracklet Affinity Networks (STAN), is presented, which is able to bi-directionally simulate the unseen dynamics of actions, comprehensively models the long-term action dependences, and sequentially estimates their affinity. Such hierarchical association is finally solved as a minimum-cost network flow problem. We extensively evaluate the proposed approach on the APIDIS, NCAA Basketball and VolleyTrack (newly collected) databases, and the experimental results show its advantages.