4.6 Article

A Declarative Metamorphic Testing Framework for Autonomous Driving

Journal

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
Volume 49, Issue 4, Pages 1964-1982

Publisher

IEEE COMPUTER SOC
DOI: 10.1109/TSE.2022.3206427

Keywords

Testing; Autonomous vehicles; Predictive models; Roads; Ontologies; Vehicles; Natural languages; Metamorphic testing; Autonomous driving; testing

Ask authors/readers for more resources

Autonomous driving has attracted much attention, but safety concerns arise due to fatal accidents caused by autonomous vehicles. This paper presents a rule-based metamorphic testing framework called RMT to detect potential issues in autonomous driving models.
Autonomous driving has gained much attention from both industry and academia. Currently, Deep Neural Networks (DNNs) are widely used for perception and control in autonomous driving. However, several fatal accidents caused by autonomous vehicles have raised serious safety concerns about autonomous driving models. Some recent studies have successfully used the metamorphic testing technique to detect thousands of potential issues in some popularly used autonomous driving models. However, prior study is limited to a small set of metamorphic relations, which do not reflect rich, real-world traffic scenarios and are also not customizable. This paper presents a novel declarative rule-based metamorphic testing framework called RMT. RMT provides a rule template with natural language syntax, allowing users to flexibly specify an enriched set of testing scenarios based on real-world traffic rules and domain knowledge. RMT automatically parses human-written rules to metamorphic relations using an NLP-based rule parser referring to an ontology list and generates test cases with a variety of image transformation engines. We evaluated RMT on three autonomous driving models. With an enriched set of metamorphic relations, RMT detected a significant number of abnormal model predictions that were not detected by prior work. Through a large-scale human study on Amazon Mechanical Turk, we further confirmed the authenticity of test cases generated by RMT and the validity of detected abnormal model predictions.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.6
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available