Journal
COMPUTER VISION, ECCV 2022, PT IX
Volume 13669, Issue -, Pages 139-158Publisher
SPRINGER INTERNATIONAL PUBLISHING AG
DOI: 10.1007/978-3-031-20077-9_9
Keywords
Object detection; Multimodal detection; Infrared; hermal; Probabilistic model; Ensembling; Multimodal fusion; Uncertainity
Funding
- CMU Argo AI Center for Autonomous Vehicle Research
Ask authors/readers for more resources
This paper investigates multimodal object detection with RGB and thermal cameras and proposes a probabilistic ensembling technique, ProbEn, that effectively fuses the detections from different modalities. It shows significant improvement in multimodal detection even when the conditional independence assumption is not met.
Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs). Motivated by AVs that operate in both day and night, we study multimodal object detection with RGB and thermal cameras, since the latter provides much stronger object signatures under poor illumination. We explore strategies for fusing information from different modalities. Our key contribution is a probabilistic ensembling technique, ProbEn, a simple nonlearned method that fuses together detections from multi-modalities. We derive ProbEn from Bayes' rule and first principles that assume conditional independence across modalities. Through probabilistic marginalization, ProbEn elegantly handles missing modalities when detectors do not fire on the same object. Importantly, ProbEn also notably improves multimodal detection even when the conditional independence assumption does not hold, e.g., fusing outputs from other fusion methods (both off-the-shelf and trained in-house). We validate ProbEn on two benchmarks containing both aligned (KAIST) and unaligned (FLIR) multimodal images, showing that ProbEn outperforms prior work by more than 13% in relative performance!
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available