Exploring granular test coverage and its evolution with matrix visualizations

Context: Current software-development tools that are used in practice make understanding the test execution of software difficult, for both granular tasks (e.g., answering questions such as, which test cases execute this method?) and global tasks (e.g., answering questions such as, what is the proportion of unit tests to system tests?). Current tools typically support local, file-based views of a project's test suite and its execution data, and rarely offer a global overview. Even more rarely, do they provide access to historical information of this nature. Such global overviews can provide a larger context for a method's execution by test cases; help identify other similar, or related methods; and even reveal similarity between individual tests. Objective: This work approaches such challenges with a novel, interactive, matrix-based visual interface that provides a global overview of a software project's test suite, specifically in the context of the methods available in the project's codebase. Through a series of interactive functions to sort, filter, query, and explore a testmatrix visualization, we evaluate how developers can effectively answer questions about their project's test suite, and the code executed by such tests.Method: We built a dynamic test-suite analysis and software-visualization tool that implements our designed interface to address the challenges of understanding the testing of software systems. With this implementation, we conducted a user study of 20 software developers to assess their ability to understand and report test execution information and measured accuracy and time. Additionally, we present a series of case studies to demonstrate a number of insights that our tool reveals.Results: Our evaluations, performed on 26 real-world software systems, show that the interactive visualization assisted developers to answer questions about software tests and the code they execute. Further, the visualization consistently outperforms traditional development tools, both in accuracy and time taken to complete software-engineering tasks.Conclusion: Global-overview test matrices offer novel perspectives on test-suite composition, which can guide software development and testing practices.

Automated summary

Current software-development tools make it difficult to understand the test execution of software, both for granular tasks (e.g., identifying test cases for a specific method) and global tasks (e.g., determining the proportion of unit tests to system tests). Existing tools lack global overview and historical information. This paper proposes a novel, interactive, matrix-based visual interface to address these challenges and provides a user study and case studies to demonstrate its effectiveness.