The reproducibility crisis meets stock assessment science: Sources of inadvertent bias in the stock assessment prioritization and review process

The broader scientific community is struggling with a reproducibility crisis brought on by numerous factors, including p-hacking or selective reporting that may increase the rate of false positives or generate misleading effect size estimates from meta-analyses. This results when multiple modeling approaches or statistical tests may be brought to bear on the same problem, and there are pressures or rewards for finding significant results. Fisheries science is unlikely to be immune to this problem, with numerous opportunities for bias to inadvertently enter into the process through the prioritization of stocks for assessment, decisions about competing model approaches or data treatments within complex assessment models, and decisions about whether to adopt as-sessments for management after they are reviewed. I present a simple simulation model of a system where many assessments are performed each management cycle for a multi-stock fishery, and show how asymmetric selection of assessments for extra scrutiny or re-assessment within a cycle can turn a process generating unbiased advice on fishing limits into one that is biased high. I show similar results when sequential assessments receive extra scrutiny if they show large proportional decreases in catch limits compared to a prior assessment for the same stock, especially if there are only small changes in true stock size or status over the interval between assessments. The level of bias introduced by a plausible level of asymmetric scrutiny is unlikely to fundamentally undermine scientific advice, but may be sufficient to compromise the nominal overfishing probabilities used in a common framework for accommodating uncertainty, and introduce a level of bias comparable to the difference between buffers corresponding to commonly-applied levels of risk tolerance.

Automated summary

The broader scientific community is facing a reproducibility crisis due to various factors, including selective reporting and p-hacking, which may lead to false positive results and misleading effect size estimates. Fisheries science is also vulnerable to this issue, as bias can inadvertently enter the process through various decision-making steps. A simulation model demonstrates how asymmetric scrutiny of assessments or re-assessments can introduce bias into the advisory process for fishing limits in a multi-stock fishery.