Endowing data-driven models with rejection ability: Out-of-distribution detection and confidence estimation for black-box models of building energy systems

As one of the most popular modeling methods, data-driven methods would fail easily when the online data are outside the scope of training data. To tackle this problem, this paper proposed a novel concept named as the rejection ability of data-driven models. Its main idea is rejecting the unreliable predictions based on the threshold which can be either calculated from out-of-distribution detection model or the confidence score. To compare with traditional data-driven models, three evaluation metrics are derived from the rejection table to quantify model performance. The usefulness of rejection method is validated through the fault detection and diagnosis problem of chillers. The ASHARE RP-1043 dataset is adopted to conduct the data experiments where the in -distribution performance and out-of-distribution performance of three task models, including RF, ET and ANN, and their combinations with four rejection methods, including AE-MSE, AE-MAE, Entropy and Max probability, are comprehensively investigated. The experimental results indicate that the model performance improves for a large margin with the introduction of rejection method. The optimal combination is ET-Entropy model, where its threshold selection is discussed for different application scenarios. The proposed rejection methods might provide a solid foundation for the real-world application of data-driven models.

Automated summary

This paper proposes a concept called the rejection ability of data-driven models to address the issue of online data being outside the scope of training data. Experimental results show that the introduction of rejection methods significantly improves model performance.