Evaluation of probabilistic prediction systems for a scalar variable

A systematic study is performed of a number of scores that can be used for objective validation of probabilistic prediction of scalar variables: Rank Histograms, Discrete and Continuous Ranked Probability Scores (DRPS and CRPS, respectively). The reliability-resolution-uncertainty decomposition, defined by Murphy for the DRPS, and extended here to the CRPS, is studied in detail. The decomposition is applied to the results of the Ensemble Prediction Systems of the European Centre for Medium-range Weather Forecasts and the National Centers for Environmental Prediction. Comparison is made with the decomposition of the CRPS defined by Hersbach. The possibility of determining an accurate reliability-resolution decomposition of the RPSs is severely limited by the unavoidably (relatively) small number of available realizations of the prediction system. The Hersbach decomposition may be an appropriate compromise between the competing needs for accuracy and practical computability.