Unsupervised anomaly detection ensembles using item response theory

Ensemble learning combines many algorithms or models to obtain better predictive perfor-mance. Ensembles have produced the winning algorithm in competitions such as the Netflix Prize. They are used in climate modeling and relied upon to make daily forecasts. Constructing an ensemble from a heterogeneous set of unsupervised anomaly detection methods presents challenges because the class labels or the ground truth is unknown. Thus, traditional ensemble techniques that use the class labels cannot be used for this task. We use Item Response Theory (IRT) - a class of models used in educational psychomet-rics - to construct an unsupervised anomaly detection ensemble. IRT's latent trait compu-tation lends itself to anomaly detection because the latent trait can be used to uncover the hidden ground truth. Using a novel IRT mapping to the anomaly detection problem, we construct an ensemble that can downplay noisy, non-discriminatory methods and accentu-ate sharper methods. We demonstrate the effectiveness of the IRT ensemble using two real data repositories and show that it outperforms other ensemble techniques. We find that the IRT ensemble performs well even if the set of anomaly detection methods have low cor-relation values.(c) 2021 Elsevier Inc. All rights reserved.

Automated summary

Ensemble learning combines multiple algorithms or models to improve predictive performance and is used in various fields. This study introduces the use of Item Response Theory (IRT), a model used in educational psychometrics, to construct an unsupervised anomaly detection ensemble. The effectiveness of the IRT ensemble is demonstrated and shown to outperform other ensemble techniques, even with low correlation values among the anomaly detection methods.