A complementary dual-backbone transformer extracting and fusing weak cues for object detection in extremely dark videos

Reliable object detection under dark environment is of wide applications but severely challenged by heavy noise washing out informative features and uneven radiance caused by nighttime illuminations. These unique features of dark videos would largely degenerate the performance of existing detectors. To address this issue, specially designed algorithms being able to extract and fuse the weak features buried in the low-quality videos are of vital importance. Bearing these in mind, we propose illumination-aware spatio-temporal feature fusion modules for low-light video object detection and implement a Dark Video Detector under a TRansformer network structure, dubbed as DVD-TR. Firstly, we use a dual-backbone Transformer to extract separate complementary features and fuse them to strengthen the network's feature extraction capability. Secondly, we incorporate a spatio-temporal sampling mechanism to aggregate features from multiple frames, which can enhance detection accuracy in dark videos. Thirdly, we use a small encoder-decoder network to obtain irradiance distribution which is further incorporated for illumination-aware feature fusion. Extensive experiments on large-scale multi-illuminance dark video benchmark show that DVD-TR outperforms state-of-the-art video detectors by a large margin and validate the effectiveness of the proposed approach.

Automated summary

Reliable object detection in dark environments is severely challenged by noise and uneven radiance. To address this, we propose illumination-aware spatio-temporal feature fusion modules for low-light video object detection under a TRansformer network structure. Extensive experiments validate the effectiveness of our approach and demonstrate that DVD-TR outperforms state-of-the-art video detectors on a large-scale multi-illuminance dark video benchmark.