Performance Evaluation Indicator (PEI): A new paradigm to evaluate the competence of machine learning classifiers in predicting rockmass conditions

To illustrate an unprejudiced comparison among machine learning classifiers established on proprietary data-bases, and to guarantee the validity and robustness of these classifiers, a Performance Evaluation Indicator (PEI) and the corresponding failure criterion are proposed in this study. Three types of machine learning classifiers, including the strictly binary classifier, the normal multiclass classifier and the misclassification cost-sensitive classifier, are trained on four datasets recorded from a water drainage TBM project. The results indicate that: (1) the PEI successfully compares the competence of classifiers under different scenarios by isolating the effects of different overlapping-degree of rockmass classes, and (2) the cost-sensitive algorithm is warranted to classify rockmasses when the ratio of inter-class classes is more than 8:1. The contributions of this research are to fill the gap in performance evaluations of a classifier for imbalanced training data, and to identify the best situation to apply this classifier.

Automated summary

This study proposes a Performance Evaluation Indicator (PEI) and corresponding failure criterion to compare machine learning classifiers on proprietary databases. The research reveals that a cost-sensitive algorithm is effective for classifying rockmasses in scenarios with imbalanced class ratios.