PREDICTION IN THE PRESENCE OF RESPONSE-DEPENDENT MISSING LABELS

In various settings, limitations of sensing technologies or other sampling mechanisms result in missing labels, where the likelihood of a missing label is an unknown function of the data. For example, satellites used to detect forest fires cannot sense fires below a certain size threshold. In such cases, training datasets consist of positive and pseudo-negative observations (true negatives or undetected positives with small magnitudes). We develop a new methodology and non-convex algorithm which jointly estimates the magnitude and occurrence of events, utilizing prior knowledge of the detection mechanism. We provide conditions under which our model is identifiable. We prove that even though our approach leads to a non-convex objective, any local minimizer has an optimal statistical error (up to a log term) and the projected gradient descent algorithm has geometric convergence rates. We demonstrate on both synthetic data and a California wildfire dataset that our method outperforms existing state-of-the-art approaches.

Automated summary

The study introduces a new methodology and algorithm that can jointly estimate the magnitude and occurrence of events in the presence of missing labels. It is proven that the proposed method exhibits optimal statistical error even with a non-convex objective, and the gradient descent algorithm used has geometric convergence rates. Performance evaluation on synthetic data and a California wildfire dataset shows that the proposed method outperforms existing state-of-the-art approaches.