Performance-Driven Metamorphic Testing of Cyber-Physical Systems

Cyber-physical systems (CPSs) are a new generation of systems, which integrate software with physical processes. The increasing complexity of these systems, combined with the uncertainty in their interactions with the physical world, makes the definition of effective test oracles especially challenging, facing the well-known test oracle problem. Metamorphic testing has shown great potential to alleviate the test oracle problem by exploiting the relations among the inputs and outputs of different executions of the system, so-called metamorphic relations (MRs). In this article, we propose an MR pattern called PV for the identification of performance-driven MRs, and we show its applicability in two CPSs from different domains, which are automated navigation systems and elevator control systems. For the evaluation, we assessed the effectiveness of this approach for detecting failures in an open-source simulation-based autonomous navigation system, as well as in an industrial case study from the elevation domain. We derive concrete MRs based on the PV pattern for both case studies, and we evaluate their effectiveness with seeded faults. Results show that the approach is effective at detecting over 88% of the seeded faults, while keeping the ratio of FPs at 4% or lower.

Automated summary

Cyber-physical systems (CPSs) are a new generation of systems that integrate software with physical processes. Metamorphic testing has shown great potential for alleviating the test oracle problem by utilizing the relationships among the inputs and outputs of different system executions. This article proposes an MR pattern (PV) for identifying performance-driven MRs and evaluates its effectiveness in two CPSs from different domains.