A fast RetinaNet fusion framework for multi-spectral pedestrian detection

At present, the mainstream visible pedestrian detector is easily affected by the ambient lighting, the complex background, and the pedestrians distance. While the infrared images (IR) can compensate for the defects of visible images because of its insensitivity to illumination conditions. Based on the Deep Convolutional Neural Network (DCNN), we proposed a multispectral pedestrian detector that combines visual-optical (VIS) image and infrared (IR) image. We carefully designed three DCNN fusion architectures to study the better fusion stages of the two-branch DCNN. In addition, we compared the three fusion strategies and found that the sum fusion strategy showed better performance to our multispectral detector. Our multispectral pedestrian detectors are more adaptable to the around-the-clock applications such as autonomous driving and unattended monitoring, by testing on the public multispectral benchmark dataset KAIST, our best fusion architectures achieved a log-average miss rate of 27.60% comparable to the state-of-the-art detector, but with half the runtime.