Evaluation of Probabilistic Project Cost Estimates

Evaluation of cost estimates should be fair and give incentives for accuracy. These goals, we argue, are challenged by a lack of precision in what is meant by a cost estimate and the use of evaluation measures that do not reward the most accurate cost estimates. To improve the situation, we suggest the use of probabilistic cost estimates and propose guidelines on how to evaluate such estimates. The guidelines emphasize the importance of a match between the type of cost estimate provided by the estimators and the chosen cost evaluation measure, and the need for an evaluation of both the calibration and the informativeness of the estimates. The feasibility of the guidelines is exemplified in an analysis of a set of 69 large Norwegian governmental projects. The evaluation indicated that the projects had quite accurate and unbiased P50 estimates and that the prediction intervals were reasonably well-calibrated. It also showed that the cost prediction intervals were noninformative with respect to differences in cost uncertainty and, consequently, not useful to identify projects with higher cost uncertainty. The results demonstrate the usefulness of applying the proposed cost estimation evaluation guidelines.

Automated summary

Evaluation of cost estimates should be fair and incentivize accuracy. However, current evaluation measures fail to reward the most accurate cost estimates. To address this issue, we propose the use of probabilistic cost estimates and provide guidelines for evaluating such estimates. An analysis of 69 large Norwegian governmental projects demonstrates the accuracy and lack of bias in the P50 estimates, as well as the calibration of prediction intervals. However, the cost prediction intervals do not provide informative differences in cost uncertainty, thus limiting their usefulness.