No Free Lunch when Estimating Simulation Parameters

In this paper, we have estimated the parameters of 41 simulation models to find which of 9 estimation algorithms performs better. Unfortunately, no single algorithm was the best for all or even most of the models. Rather, five main results emerge from this research. First, each algorithm was the best estimator for at least one parameter. Second, the best estimation algorithm varied not only between models but even between parameters of the same model. Third, each estimation algorithm failed to estimate at least one identifiable parameter. Fourth, choosing the right algorithm improved estimation performance by more than quadrupling the number of model runs. Fifth, half of the agent-based models tested could not be fully identified. We therefore argue that the testing performed here should be done in other applied work and to facilitate this we would like to share the R package freelunch.

Automated summary

Each estimation algorithm has its strengths and weaknesses, and there is not a single best choice for all or most models. Selecting the right algorithm is crucial for improving estimation performance, but some identifiable parameters still cannot be accurately estimated.