Joint spatio-temporal modeling for visual tracking

Similarity-based approaches have made significant progress in visual object tracking (VOT). Although these methods work well in simple scenes, they ignore the continuous spatio-temporal connection of the object in the video sequence. For this reason, tracking by spatial matching solely can lead to tracking failures because of distractors and occlusion. In this paper, we propose a spatio-temporal joint-modeling tracker named STTrack which implicitly builds continuous connections between the temporal and spatial aspects of the sequence. Specifically, we first design a time-sequence iteration strategy (TSIS) to concentrate on the temporal connection of the object in the video sequence. Then, we propose a novel spatial temporal interaction Transformer network (STIN) to capture the spatio-temporal correlation of the object between frames. The proposed STIN module is robust in object occlusion because it explores the dynamic state change dependencies of the object. Finally, we introduce a spatio-temporal query to suppress distractors by iteratively propagating the target prior. Extensive experiments on six tracking benchmark datasets demonstrate that the proposed STTrack achieves excellent performance while operating in real-time. The code is publicly available at https://github.com/nubsym/STTrack.

Automated summary

In this paper, a spatio-temporal joint-modeling tracker named STTrack is proposed for continuous object tracking in video sequences. The proposed method concentrates on the temporal connection of the object by employing a time-sequence iteration strategy (TSIS) and captures the spatio-temporal correlation of the object between frames using a novel spatial temporal interaction Transformer network (STIN). Experimental results demonstrate that STTrack achieves excellent performance on six tracking benchmark datasets while operating in real-time.