Challenges in benchmarking metagenomic profilers

Many computational tools for metagenomic profiling have been developed, with different algorithms and features. This analysis shows that, when comparing these tools, the distinction of different types of relative sequence abundance should be taken into consideration. Accurate microbial identification and abundance estimation are crucial for metagenomics analysis. Various methods for classification of metagenomic data and estimation of taxonomic profiles, broadly referred to as metagenomic profilers, have been developed. Nevertheless, benchmarking of metagenomic profilers remains challenging because some tools are designed to report relative sequence abundance while others report relative taxonomic abundance. Here we show how misleading conclusions can be drawn by neglecting this distinction between relative abundance types when benchmarking metagenomic profilers. Moreover, we show compelling evidence that interchanging sequence abundance and taxonomic abundance will influence both per-sample summary statistics and cross-sample comparisons. We suggest that the microbiome research community pay attention to potentially misleading biological conclusions arising from this issue when benchmarking metagenomic profilers, by carefully considering the type of abundance data that were analyzed and interpreted and clearly stating the strategy used for metagenomic profiling.

Automated summary

A variety of computational tools exist for metagenomic profiling, each with distinct algorithms and features. It is crucial to consider the distinction between different types of relative sequence abundance when comparing these tools. Neglecting this distinction can lead to misleading conclusions when benchmarking metagenomic profilers, impacting both per-sample summary statistics and cross-sample comparisons. The microbiome research community should carefully consider the type of abundance data analyzed and clearly state the profiling strategy used to avoid potentially misleading biological conclusions.