Attention-Enhanced Cross-Modal Localization Between Spherical Images and Point Clouds

Visual localization plays an important role for intelligent robots and autonomous driving, especially when the accuracy of GNSS is unreliable. Recently, camera localization in LiDAR maps has attracted more and more attention for its low cost and potential robustness to illumination and weather changes. However, the commonly used pinhole camera has a narrow field-of-view, leading to limited information compared with the omnidirectional LiDAR data. To overcome this limitation, we focus on correlating the information of 360 degrees spherical images to point clouds, proposing an end-to-end learnable network to conduct cross-modal visual localization by establishing similarity in high-dimensional feature space. Inspired by the attention mechanism, we optimize the network to capture the salient feature for comparing images and point clouds. We construct several 2-D-3-D sequences containing 360 degrees spherical images and the corresponding point clouds based on the KITTI-360 dataset and conduct extensive experiments. The evaluation results demonstrate the effectiveness of our approach. The source code and dataset are released at https://github.com/Zhaozhpe/AE-CrossModal.

Automated summary

Visual localization plays a crucial role in intelligent robots and autonomous driving, especially when GNSS is unreliable. To address the limited information provided by the commonly used pinhole camera with a narrow field-of-view, we propose an end-to-end learnable network that correlates 360 degrees spherical images with point clouds for cross-modal visual localization. Inspired by the attention mechanism, we optimize the network to capture salient features for comparing images and point clouds. Our approach is evaluated on the KITTI-360 dataset and achieves promising results.