RIbench: A Proposed Benchmark for the Standardized Evaluation of Indirect Methods for Reference Interval Estimation

Background Indirect methods leverage real-world data for the estimation of reference intervals. These constitute an active field of research, and several methods have been developed recently. So far, no standardized tool for evaluation and comparison of indirect methods exists. Methods We provide RIbench, a benchmarking suite for quantitative evaluation of any existing or novel indirect method. The benchmark contains simulated test sets for 10 biomarkers mimicking routine measurements of a mixed distribution of non-pathological (reference) values and pathological values. The non-pathological distributions represent 4 common distribution types: normal, skewed, heavily skewed, and skewed-and-shifted. To identify strengths and weaknesses of indirect methods, test sets have varying sample sizes and pathological distributions differ in location, extent of overlap, and fraction. For performance evaluation, we use an overall benchmark score and sub-scores derived from absolute z-score deviations between estimated and true reference limits. We illustrate the application of RIbench by evaluating and comparing the Hoffmann method and 4 modern indirect methods -TML (Truncated-Maximum-Likelihood), kosmic, TMC (Truncated-Minimum-Chi-Square), and refineR- against one another and against a nonparametric direct method (n = 120). Results For the modern indirect methods, pathological fraction and sample size had a strong influence on the results: With a pathological fraction up to 20% and a minimum sample size of 5000, most methods achieved results comparable or superior to the direct method. Conclusions We present RIbench, an open-source R-package, for the systematic evaluation of existing and novel indirect methods. RIbench can serve as a tool for enhancement of indirect methods, improving the estimation of reference intervals.

Automated summary

This study developed a benchmarking suite called RIbench for evaluating and comparing the performance of indirect methods in estimating reference intervals. Through the evaluation and comparison of several modern indirect methods, the results showed that these methods outperformed the direct method when the pathological fraction was below 20% and the sample size exceeded 5000.