Journal
IEEE ROBOTICS AND AUTOMATION LETTERS
Volume 7, Issue 3, Pages 7415-7422Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/LRA.2022.3183931
Keywords
Computer vision for transportation; deep learning for visual perception; representation learning
Categories
Funding
- Deutsche Forschungsgemeinschaft [BU 865/10-2]
Ask authors/readers for more resources
This paper proposes a self-supervised approach that utilizes audio-visual cues to detect moving vehicles in videos. By employing contrastive learning with corresponding pairs of images and recorded audio, the approach achieves accurate detections of vehicles without the need for manual annotations.
Robust detection of moving vehicles is a critical task for any autonomously operating outdoor robot or self-driving vehicle. Mast modern approaches for solving this task rely on training image-based detectors using large-scale vehicle detection datasets such as nuScenes or the Waymo Open Dataset. Providing manual annotations is an expensive and laborious exercise that does not scale well in practice. To tackle this problem, we propose a self-supervised approach that leverages audio-visual cues to detect moving vehicles in videos. Our approach employs contrastive learning for localizing vehicles in images from corresponding pairs of images and recorded audio. In extensive experiments carried out with a real-world dataset, we demonstrate that our approach provides accurate detections of moving vehicles and does not require manual annotations. We furthermore show that our model can he used as a teacher to supervise an audio-only detection model. This student model is invariant to illumination changes and thus effectively bridges the domain gap inherent to models leveraging exclusively vision as the predominant modality.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available