How to measure uncertainty in uncertainty sampling for active learning

Various strategies for active learning have been proposed in the machine learning literature. In uncertainty sampling, which is among the most popular approaches, the active learner sequentially queries the label of those instances for which its current prediction is maximally uncertain. The predictions as well as the measures used to quantify the degree of uncertainty, such as entropy, are traditionally of a probabilistic nature. Yet, alternative approaches to capturing uncertainty in machine learning, alongside with corresponding uncertainty measures, have been proposed in recent years. In particular, some of these measures seek to distinguish different sources and to separate different types of uncertainty, such as the reducible (epistemic) and the irreducible (aleatoric) part of the total uncertainty in a prediction. The goal of this paper is to elaborate on the usefulness of such measures for uncertainty sampling, and to compare their performance in active learning. To this end, we instantiate uncertainty sampling with different measures, analyze the properties of the sampling strategies thus obtained, and compare them in an experimental study.

Automated summary

Various strategies for active learning have been proposed in the machine learning literature, with uncertainty sampling being one of the most popular approaches. While traditional measures of uncertainty are probabilistic, alternative methods for capturing uncertainty in machine learning have been introduced in recent years. This paper aims to examine the usefulness of such measures for uncertainty sampling and compare their performance in active learning through an experimental study.