An online semantic mapping system for extending and enhancing visual SLAM

We present a real-time semantic mapping approach for mobile vision systems with a 2D to 3D object detection pipeline and rapid data association for generated landmarks. Besides the semantic map enrichment the associated detections are further introduced as semantic constraints into a simultaneous localization and mapping (SLAM) system for pose correction purposes. This way, we are able generate additional meaningful information that allows to achieve higher-level tasks, while simultaneously leveraging the view-invariance of object detections to improve the accuracy and the robustness of the odometry estimation. We propose trackless of locally associated object observations to handle ambiguous and false predictions and an uncertainty-based greedy association scheme for an accelerated processing time. Our system reaches real-time capabilities with an average iteration duration of 65 ms and is able to improve the pose estimation of a state-of-the-art SLAM by up to 68% on a public dataset. Additionally, we implemented our approach as a modular ROS package that makes it straightforward for integration in arbitrary graph-based SLAM methods.

Automated summary

We present a real-time semantic mapping approach for mobile vision systems that involves a 2D to 3D object detection pipeline and rapid data association. The associated detections are introduced into a SLAM system for pose correction, enabling the generation of additional meaningful information to achieve higher-level tasks. By leveraging the view-invariance of object detections, the accuracy and robustness of odometry estimation are improved.