End-to-End Detection of a Landing Platform for Offshore UAVs Based on a Multimodal Early Fusion Approach

A perception module is a vital component of a modern robotic system. Vision, radar, thermal, and LiDAR are the most common choices of sensors for environmental awareness. Relying on singular sources of information is prone to be affected by specific environmental conditions (e.g., visual cameras are affected by glary or dark environments). Thus, relying on different sensors is an essential step to introduce robustness against various environmental conditions. Hence, a perception system with sensor fusion capabilities produces the desired redundant and reliable awareness critical for real-world systems. This paper proposes a novel early fusion module that is reliable against individual cases of sensor failure when detecting an offshore maritime platform for UAV landing. The model explores the early fusion of a still unexplored combination of visual, infrared, and LiDAR modalities. The contribution is described by suggesting a simple methodology that intends to facilitate the training and inference of a lightweight state-of-the-art object detector. The early fusion based detector achieves solid detection recalls up to 99% for all cases of sensor failure and extreme weather conditions such as glary, dark, and foggy scenarios in fair real-time inference duration below 6 ms.

Automated summary

A perception module is crucial for a modern robotic system, with the most common sensor choices being vision, radar, thermal, and LiDAR for environmental awareness. Relying on a single source of information is susceptible to specific environmental conditions, so using different sensors is essential for robustness. This paper proposes an early fusion module that combines visual, infrared, and LiDAR modalities to detect offshore maritime platforms for UAV landing. The early fusion-based detector achieves high detection recalls of up to 99% in all cases of sensor failure and extreme weather conditions, with an inference duration below 6 ms.