MetaLiDAR: Automated metamorphic testing of LiDAR-based autonomous driving systems

Recent advances in artificial intelligence technology and perception components have promoted the rapid development of autonomous vehicles. However, as safety-critical software, autonomous driving systems often make wrong judgments, seriously threatening human and property safety. LiDAR is one of the most critical sensors in autonomous vehicles, capable of accurately perceiving the three-dimensional information of the environment. Nevertheless, the high cost of manually collecting and labeling point cloud data leads to a dearth of testing methods for LiDAR-based perception modules. To bridge the critical gap, we introduce MetaLiDAR, a novel automated metamorphic testing methodology for LiDAR-based autonomous driving systems. First, we propose three object-level metamorphic relations for the domain characteristics of autonomous driving systems. Next, we design three transformation modules so that MetaLiDAR can generate natural-looking follow-up point clouds. Finally, we define corresponding evaluation metrics based on metamorphic relations. MetaLiDAR automatically determines whether source and follow-up test cases meet the metamorphic relations based on the evaluation metrics. Our empirical research on five state-of-the-art LiDAR-based object detection models shows that MetaLiDAR can not only generate natural-looking test point clouds to detect 181,547 inconsistent behaviors of different models but also significantly enhance the robustness of models by retraining with synthetic point clouds. We introduce MetaLiDAR, to alleviate the test oracle and test case generation problems of LiDAR-based autonomous driving systems. MetaLiDAR utilizes carefully designed metamorphic relations and object-level operation methods to automatically generate natural-looking point clouds that satisfy the physical characteristics of LiDAR and reveal inconsistent behaviors of models without test oracle.image

Automated summary

The study introduces MetaLiDAR, an automated metamorphic testing methodology for LiDAR-based autonomous driving systems. MetaLiDAR generates natural-looking point clouds and utilizes metamorphic relations and evaluation metrics for automated testing, significantly enhancing the robustness of models.