On the Reliability of the Rank Histogram

Ensemble consistency is a name for the condition that an observation being forecast by a dynamical ensemble is statistically indistinguishable from the ensemble members. This statistical indistinguishability condition is meaningful only in a multivariate sense. That is, it pertains to the joint distribution of the ensemble members and the observation. The rank histogram has been designed to assess overall ensemble consistency, but mistakenly employing it to assess only restricted aspects of this joint distribution (e.g., the climatological distribution) leads to the incorrect conclusion that the verification rank histogram is not a useful diagnostic for good behavior of ensemble forecasts. The potential confusion is analyzed in the context of an idealized multivariate Gaussian model of forecast ensembles and their corresponding observations, and it is shown that the rank histogram does correctly assess the consistency of forecast ensembles.