Reproducibility and external validity of on-farm experimental research in Africa

Agronomists have increasingly conducted experiments on-farm, in an attempt to increase the wider applicability (external validity) of their experimental findings and their relevance for agricultural development. This review assesses the way in which on-farm experimental studies address the scope or generalisability of their findings when based on a limited number of farms. A central question is how on-farm studies define the environment or research population in which the on-farm trial findings are valid, or are valuable for. Such an assessment is, of course, conditional on the (internal) validity of the experimental findings. We therefore first analyse how authors of on-farm experimental studies describe the factors that may shape experimental outcomes. As agronomic experiments often use 'yield' as dependent variable to assess treatment effects, we developed a procedure to score studies on their descriptions of yield-determining factors. Although experimental validity principally rests upon the reproducibility of the experiment and its findings, we found that on the basis of the information provided in published on-farm experimental studies, it is often difficult or impossible to reproduce the experimental design. Nutrient management, weed management and crop information are best described, whereas land preparation, field history and management of pests and water are rarely described. Further, on-farm experimental studies often compare treatments to a 'farmer practice' reference or control treatment which is assumed to be widely and uniformly practiced and known to the reader. The wider applicability or external validity is often poorly addressed in the reviewed studies. Most do not explicitly define the research population and/or environment in which (they expect) the experimental findings to work. Academic textbooks on agronomic experimentation are remarkably silent on both the internal and external validity of on-farm experimentation. We therefore argue for more systematic investigations and descriptions of the research population and settings to which on-farm experimental studies seek to generalise their findings.