Evaluating indices of conservation success: a comparative analysis of outcome- and output-based indices

Conservation funders require methods by which to evaluate the relative success of projects within their portfolios. One approach is to develop robust indices of success that are consistent between projects and evaluators. We used three contrasting indices to evaluate factors contributing to success of projects funded by the UK Government's Darwin Initiative. The indices were: Darwin Outputs (related to the Darwin Initiative's own method of evaluating the success of the projects it supports), Impact Assessment (based on the method developed by Mace et?al. for evaluating the success of projects supported by zoos) and a Ranked Outcomes (a qualitatively ranked outcome index). We evaluated the internal consistency of the indices by comparing the assessments of multiple independent scorers. We assessed their robustness by checking for differences between indices and assessors in the success level assigned to a given project. We then used mixed effects models to analyse the factors contributing to project success, as expressed by each index, and compared the factors highlighted as important by each index. Although there were systematic differences between scorers, relative rankings between scorers were consistent. The indices were in fair agreement as to project success ranks, although the success ranks assigned by subjective ranked outcome- and output-based indices were more consistent between assessors than the impact assessment index. Higher levels of funding led to projects receiving consistently higher success scores. Other variables varied in their importance between indices, although metrics of education were consistently important. This study shows that it is possible to develop robust outcome-based indices of conservation success for comparison of projects within a funder's portfolio, although the nuances picked up by different indices suggest a need for multiple indices to capture different facets of success. We also highlight the need for thorough testing of the robustness of success indices before widespread adoption.