Exceedance Probability Score: A Novel Measure for Comparing Probabilistic Predictions

Accurate prediction of exceedance probabilities is important in many applications. For example, in process planning and control, engineers should anticipate the risk that a product fails to meet its specification limits. Statistical comparison between candidate probability prediction methods is commonly performed using scoring rules, like the continuous ranked probability score (CRPS) and the logarithm score (LogS). In this work, a new scoring rule, the exceedance probability score, is proposed. The experiments in simulated and real industrial data show that the new scoring rule is useful in comparing and testing differences in the predictive accuracy of competitive probabilistic predictions in regression setting. The proposed scoring rule have some similarities with CRPS and LogS, but is more directly connected to the accuracy in the prediction of exceedance probabilities.