Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model

Objectives: To explore how factors relating to grades and grading affect the correctness of choices that grant review panels make among submitted proposals. To identify interventions in panel design that may be expected to increase the correctness of choices. Method: Experimentation with an empirically-calibrated computer simulation model of panel review. Model parameters are set in accordance with procedures at a national science funding agency. Correctness of choices among research proposals is operationalized as agreement with the choices of an elite panel. Conclusions: The simulation model generates several hypotheses to guide further research. Increasing the number of grades used by panel members increases the correctness of simulated choices among submitted proposals. Collective decision procedures giving panels a greater capacity for discriminating among proposals also increase correctness. Surprisingly, differences in grading standards among panel members do not appreciably decrease correctness.

Automated summary

This study aims to explore how factors relating to grades and grading affect the correctness of choices made by review panels among submitted proposals, and identify interventions in panel design that can increase the correctness of choices. The results of the experiment indicate that increasing the number of grades used by panel members and giving panels a greater capacity for discriminating among proposals can improve the correctness of choices, while differences in grading standards among panel members do not significantly decrease correctness.