Investigating bias in impact observation sources and implications for impact-based forecast evaluation

Impact-based forecasts and warnings (IbFWs) are used by meteorological agencies to forecast the likelihood and severity of impacts resulting from hydrometeorological hazards. Despite progress developing IbFW approaches the evaluation of IbFWs performance is challenging, partly due to the lack of impact observations. Impact data may provide the observations required to evaluate IbFWs, and publicly accessible impact data are available from a number of diverse sources. How-ever, differences in the purpose, format, and method of collection across agencies has led to a fragmented and inconsistent picture of impacts to date. Furthermore, impact data can suffer from bias. To understand the implications of using impact data for IbF evaluation, we interrogate a range of publicly available impact data sources including structured databases, disaster reports, and dedicated web and news sites. Bias within publicly available impact data was assessed, and the implication of bias on IbFW evaluation considered. We find that each data source investigated has bias, although the type and magnitude of bias varies depending on the data source. Observed biases affect the coverage, severity, timing, geographic scale, and accuracy of impact observa-tions. Furthermore, inconsistences in hazard terminology, poor adoption of unique event identi-fiers, and the inability to relate impact observations to specific hazardous events presents chal-lenges when using impact data for IbFW evaluation. Despite these challenges, using a range of data sources together can provide a more holistic observation of the impacts of hydrometeorolog-ical events. Consolidating impact observations from a range of sources may produce the data re-quired for IbFW evaluation.

Automated summary

Impact-based forecasts and warnings (IbFWs) are used by meteorological agencies to predict the likelihood and severity of impacts from hydrometeorological hazards. Evaluating the performance of IbFWs is challenging due to the lack of impact observations and biases in publicly available impact data. Different data sources have varying types and magnitudes of bias, affecting the coverage, severity, timing, geographic scale, and accuracy of impact observations. Despite these challenges, using a range of data sources can provide a more comprehensive understanding of hydrometeorological events.