Better sure than safe? Over-confidence in judgement based software development effort prediction intervals

The uncertainty of a software development effort estimate can be indicated through a prediction interval (PI), i.e., the estimated minimum and maximum effort corresponding to a specific confidence level. For example, a project manager may be 90% confident or believe that is it very likely that the effort required to complete a project will be between 8000 and 12,000 work-hours. This paper describes results from four studies (Studies A-D) on human judgement (expert) based PIs of software development effort. Study A examines the accuracy of the PIs in real software projects. The results suggest that the PIs were generally much too narrow to reflect the chosen level of confidence, i.e., that there was a strong over-confidence. Studies B-D try to understand the reasons for the observed over-confidence. Study B examines the possibility that the over-confidence is related to type of experience or estimation process. Study C examines the possibility that the concept of confidence level is difficult to interpret for software estimators. Finally, Study D examines the possibility that there are unfortunate feedback mechanisms that reward over-confidence. (C) 2003 Elsevier Inc. All rights reserved.