Comments on MoNuSAC2020: A Multi-Organ Nuclei Segmentation and Classification Challenge

The MoNuSAC 2020 challenge was hosted at the ISBI 2020 conference, where the winners were announced. Challenge organizers, in addition to the leaderboard, released the evaluation code and visualisations of the prediction masks of the top 5 teams. This shows a very high level of transparency, and provides a unique opportunity to better understand the challenge results. Our analysis of the code and all released data, however, shows three different problems in the computation of the metric used for the official ranking: a coding mistake resulting in erroneous false positives; another resulting in missed false positives; and a problem with the metric's aggregation method. We demonstrate the errors, and confirm that the mistaken version of the code was indeed used to rank the algorithms in the challenge. Our results can be fully replicated with the code provided on GitHub.

Automated summary

The MoNuSAC 2020 challenge hosted at the ISBI 2020 conference has been analyzed, revealing three problems in the computation of the metric used for ranking. The incorrect code version was used to rank the algorithms in the challenge. The results can be replicated using the code provided on GitHub.