Instance Space Analysis of Search-Based Software Testing

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances - in the form of Instance Space Analysis (ISA) - enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.

Automated summary

Search-based software testing (SBST) is a mature area with techniques developed to tackle the challenging task of software testing. SBST techniques have been successfully applied in the industry to generate test cases for large and complex software systems. However, their effectiveness depends on the problem being addressed. This paper revisits the evaluation of SBST techniques using Instance Space Analysis (ISA) to visualize and assess their strengths and weaknesses across a broad range of problem instances from common benchmark datasets. The paper also examines the diversity and quality of benchmark datasets used in experimental evaluations.