Single-pass active learning with conflict and ignorance

In this paper, we present a new methodology for conducting active learning in a single-pass on-line learning context. Single-pass active learning can be understood as an approach for reducing the annotation effort for users and operators in on-line classification problems, in which usually the true class labels of new incoming samples are usually unknown. This reduction in effort can be achieved by selecting the most informative samples, that is, those that contribute most to improving the predictive performance of incremental classifiers. Our approach builds upon certainty-based sample selection in connection with version-space reduction. Two new reliability concepts were investigated and developed in connection with evolving fuzzy classifiers: conflict and ignorance. Conflict models the extent to which a new query point lies in the conflict region between two or more classes and therefore reflects a level of certainty in the classifier's prediction. Ignorance represents the distance of a new query point from the training samples seen so far. In extended form, it integrates the actual variability of the version space. The choice of the model architecture used for on-line classification scenarios (evolving fuzzy classifier) is clearly motivated in the paper. The results based on real-world binary and multi-class classification streaming data show that our single-pass active learning approach yields evolving classifiers whose performance is similar to that of classifiers using all samples for adaptation; however, the annotation effort in terms of the number of class label requests is reduced by up to 90 %.