A model-based approach for specifying changes in replications of empirical studies in computer Science

The need of replicating empirical studies in Computer Science is widely recognized among the research community. It is essential to report the changes of each replication to promote not only the comprehensibility of the evolution of the experimental validity across a family of studies, but also replicability itself. Unfortunately, the lack of proposals for systematic reporting of changes in replications undermines these desirable objectives. The main goal of the work presented in this article is to provide researchers in Computer Science with a systematic tool-supported approach for the specification and reporting of changes in the replications of their empirical studies. Applying Design Science Research, we have developed and validated a composite artifact consisting of (i) a metamodel that formalizes all the relevant concepts related to replications and their changes; (ii) templates and linguistic patterns that facilitate their reporting; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study that includes 9 families of empirical studies not only from Computer Science, but also from an area as different as Agrobiology , to check the external validity of our approach. The 9 families encompass 23 replication studies and a total of 92 replication changes, for which we have analyzed the suitability of our proposal. The multiple case study revealed some initial limitations of our approach related to threats to experimental validity and context variables. After several improvement iterations on the artifact, all of the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity and their corresponding visualization. Our proposal for the specification of replication changes seems to fit the needs not only of replications in Computer Science, but also in other research areas. Nevertheless, further research is needed to improve it and disseminate its use among the research community.

Automated summary

The main goal of this article is to provide a systematic tool-supported approach for the specification and reporting of changes in replications of empirical studies in Computer Science. The developed artifact includes a metamodel, templates and linguistic patterns, and a model-based software tool. A multiple case study with 9 families of empirical studies was conducted to validate the approach, revealing some initial limitations. The proposed method seems to be applicable not only in Computer Science but also in other research areas.