Optimizing Early Warning Classifiers to Control False Alarms via a Minimum Precision Constraint

Early warning prediction systems can suffer from high false alarm rates that limit utility, especially in settings with high class imbalance such as healthcare. Despite the widespread need to control false alarms, the dominant classifier training paradigm remains minimizing cross entropy, a loss function which does not treat false alarms differently than other types of mistakes. While existing efforts often try to reduce false alarms by post-hoc threshold selection after training, we suggest a comprehensive solution by changing the loss function used to train the classifier. Our proposed objective maximizes recall while enforcing a constraint requiring precision to exceed a specified value. We make our objective tractable for gradient-based optimization by developing tight sigmoidal bounds on the counts needed to compute precision and recall. Our objective is applicable to any classifier trainable via gradient descent, including linear models and neural networks. When predicting mortality risk across two large hospital datasets, we show how our method satisfies a desired constraint on false alarms while achieving better recall than alternatives.

Automated summary

Early warning prediction systems can suffer from high false alarm rates, especially in settings with high class imbalance such as healthcare. The dominant classifier training paradigm, minimizing cross entropy, does not differentiate false alarms from other types of mistakes. We propose a comprehensive solution by changing the loss function used to train the classifier, maximizing recall while enforcing a constraint on precision.