Conditional Coverage Estimation for High-Quality Prediction Intervals

Deep learning has been recently studied to generate high-quality prediction intervals (PIs) for uncertainty quantification in regression tasks, including recent applications in simulation metamodeling. The high-quality criterion requires PIs to be as narrow as possible, whilst maintaining a pre-specified level of data (marginal) coverage. However, most existing works for high-quality PIs lack accurate information on conditional coverage, which may cause unreliable predictions if it is significantly smaller than the marginal coverage. To address this problem, we propose an end-to-end framework which could output high-quality PIs and simultaneously provide their conditional coverage estimation. In doing so, we design a new loss function that is both easy-to-implement and theoretically justified via an exponential concentration bound. Our evaluation on real-world benchmark datasets and synthetic examples shows that our approach not only achieves competitive results on high-quality PIs in terms of average PI width, but also accurately estimates conditional coverage information that is useful in assessing model uncertainty.

Automated summary

Deep learning is used to generate high-quality prediction intervals (PIs) to quantify uncertainty in regression tasks, including simulation metamodeling. Most existing methods lack accurate information on conditional coverage, which may lead to unreliable predictions if it is significantly smaller than the marginal coverage. To address this, an end-to-end framework is proposed to output high-quality PIs and provide conditional coverage estimation. A new loss function is designed for implementation and theoretically justified using an exponential concentration bound. Evaluation results show competitive performance in terms of average PI width and accurate estimation of conditional coverage.