Method and model comparison by sum of ranking differences in cases of repeated observations (ties)

Significant progress has been achieved since the introduction of the new similarity measure: the sum of absolute ranking differences (SRDs) [TrAC - Trends in Anal. Chem. 29 (2010) 101-109]. Empirical evidences were accumulated about scaling, selection of the reference (benchmark) vector, cross-validation and grouping of variables (features, models, methods, etc.). The theory has been developed including the repeated observations (ties): (i) The exact theoretical distribution (null distribution) for 4 < number of objects < 9 has been calculated for SRD treatment of random numbers (it's a kind of validation, a permutation test). All possible reference vectors with ties imply different distribution. (ii) For number of objects above eight (n > 8) an approximation has been developed using the Gaussian distribution fitted on the SRD distribution given by generating of three million n-dimensional random vectors. The validity and features of the SRD methodology with ties are illustrated using two case studies: evaluation of a sensory panel and ranking of financial indicators. (C) 2013 Elsevier B.V. All rights reserved.