Transparent modeling of influenza incidence: Big data or a single data point from psychological theory?

Simple, transparent rules are often frowned upon while complex, black-box models are seen as holding greater promise. Yet in quickly changing situations, simple rules can protect against overfitting and adapt quickly. We show that the surprisingly simple recency heuristic forecasts more accurately than Google Flu Trends (GFT) which used big data analytics and a black-box algorithm. This heuristic predicts that this week's proportion of flu-related doctor visits equals the proportion from the most recent week.It is based on psychological theory of how people deal with rapidly changing situations. Other theory-inspired heuristics have outperformed big data models in predicting outcomes, such as U.S. presidential elections, or other uncertain events, such as consumer purchases, patient hospitalizations, and terrorist attacks. Heuristics are transparent, clearly communicating the underlying rationale for their predictions. We advocate taking into account psychological principles that have evolved over millennia and using these as a benchmark when testing big data models. (C) 2020 International Institute of Forecasters. Published by Elsevier B.V. All rights reserved.

Automated summary

Research shows that simple rules and theory-inspired heuristics often outperform complex big data models in forecasting accuracy. These heuristics are transparent and clearly communicate the underlying rationale for their predictions.