Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this article devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.

Automated summary

The development of autonomous vehicles has been rapid in recent years, yet achieving full autonomy poses challenges due to the complex and dynamic driving environments. The fusion of camera and LiDAR sensors using deep learning is an emerging research theme. Despite the lack of critical reviews on deep-learning-based camera-LiDAR fusion methods, recent research has focused on leveraging image and point cloud data processing for improved environmental perception and object detection.