Beyond AP: a new evaluation index for multiclass classification task accuracy

Average precision (AP) and many other related evaluation indices have been employed ubiquitously in classification tasks for a long time. However, they have defects and can hardly provide both overall evaluations and individual evaluations. In practice, we have to strike a balance between whole and individual performances to satisfy diverse demands. To this end, we propose a new index for multiclass classification tasks, named R', which is an unbiased estimator of AP. Specifically, we improve the R index by taking the numerical differences between the real labels and predicted labels of each class into consideration. We evaluate its effectiveness and robustness on the MNIST and CIFAR-10 datasets. Experimental results show that it is positively correlated with some related indices. More importantly, we can obtain both overall and individual evaluations, which can be beneficial for improving training processes and model selection. Furthermore, as an evaluation architecture, the index can be promoted to evaluate any classification task, thereby implying broad application prospects.

Automated summary

The study introduces a new metric R' for multiclass classification tasks, providing both overall and individual evaluations to improve training processes and model selection.