Unsupervised Cross-Spectrum Depth Estimation by Visible-Light and Thermal Cameras

Cross-spectrum depth estimation aims to provide a reliable depth map under variant-illumination conditions with a pair of dual-spectrum images. It is valuable for autonomous driving applications when vehicles are equipped with two cameras of different modalities. However, images captured by different-modality cameras can be photometrically quite different, which makes cross-spectrum depth estimation a very challenging problem. Moreover, the shortage of large-scale open-source datasets also retards further research in this field. In this paper, we propose an unsupervised visible light(VIS)-image-guided cross-spectrum (i.e., thermal and visible-light, TIR-VIS in short) depth-estimation framework. The input of the framework consists of a cross-spectrum stereo pair (one VIS image and one thermal image). First, we train a depth-estimation base network using VIS-image stereo pairs. To adapt the trained depth-estimation network to the cross-spectrum images, we propose a multi-scale feature-transfer network to transfer features from the TIR domain to the VIS domain at the feature level. Furthermore, we introduce a mechanism of cross-spectrum depth cycle-consistency to improve the depth estimation result of dual-spectrum image pairs. Meanwhile, we release to society a large cross-spectrum dataset with visible-light and thermal stereo images captured in different scenes. The experiment result shows that our method achieves better depth-estimation results than the compared existing methods. Our code and dataset are available on https://github.com/whitecrow1027/CrossSP_Depth.

Automated summary

In this paper, an unsupervised visible light-guided cross-spectrum depth-estimation framework is proposed. It achieves reliable depth maps under variant-illumination conditions with a pair of dual-spectrum images. Through training a depth-estimation base network, transferring features from the TIR domain to the VIS domain, and introducing a mechanism of cross-spectrum depth cycle-consistency, our method outperforms existing methods in depth estimation.