期刊
EXPERT SYSTEMS WITH APPLICATIONS
卷 229, 期 -, 页码 -出版社
PERGAMON-ELSEVIER SCIENCE LTD
DOI: 10.1016/j.eswa.2023.120461
关键词
Misclassification detection; Open-set recognition; Out-of-distribution detection; Unknown detection
Deep neural networks have achieved outstanding performance but suffer from over-confident predictions for unknown samples. Previous studies tackled this issue through specific tasks like misclassification detection or out-of-distribution detection. In this work, we propose the unknown detection task, which integrates these individual tasks, to rigorously evaluate deep neural networks' capabilities. We found that Deep Ensemble consistently outperforms other methods, but all methods are only successful for specific types of unknowns.
Deep neural networks have achieved outstanding performance over various tasks, but they have a critical issue: over-confident predictions even for completely unknown samples. Many studies have been proposed to successfully filter out these unknown samples, but they only considered narrow and specific tasks, referred to as misclassification detection, open-set recognition, or out-of-distribution detection. In this work, we argue that these tasks should be treated as fundamentally an identical problem because an ideal model should possess detection capability for all those tasks. Therefore, we introduce the unknown detection task, an integration of previous individual tasks, for a rigorous examination of the detection capability of deep neural networks on a wide spectrum of unknown samples. To this end, unified benchmark datasets on different scales were constructed and the unknown detection capabilities of existing popular methods were subject to comparison. We found that Deep Ensemble consistently outperforms the other approaches in detecting unknowns; however, all methods are only successful for a specific type of unknown. The reproducible code and benchmark datasets are available at https://github.com/daintlab/unknown-detection-benchmarks.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据