Multivariate comparison of classification performance measures

The assessment of the classification performance can be based on class indices, such as sensitivity, specificity and precision, which describe the classification results achieved on each modelled class. However, in several situations, it is useful to represent the global classification performance with a single number. Therefore, several measures have been introduced in literature to deal with this assessment, accuracy being the most known and used. These metrics have been proposed to generally face binary classification tasks and can behave differently depending on the classification scenario. In this study, different global measures of classification performances are compared by means of results achieved on an extended set of real multivariate datasets. The systematic comparison is carried out through multivariate analysis. Further investigations are then derived on specific indices to understand how the presence of unbalanced classes and the number of modelled classes can influence their behaviour. Finally, this work introduces a set of benchmark values based on different random classification scenarios. These benchmark thresholds can serve as the initial criterion to accept or reject a classification model on the basis of its performance.