Stratified random sampling for neural network test input selection

Context: Testing techniques to ensure the quality of deep neural networks (DNNs) are essential and crucial. However, the testing process can be inefficient due to a large number of test cases and the manual effort of labeling them. Recent work tackles the above challenge by selecting a small but representative subset of the tests. Such an approach allows us to quickly estimate the accuracy of a DNN with reduced effort, because only a small set of tests are to be manually labeled. However, existing approaches cannot guarantee unbiased results or provide an accurate estimation.Objectives: In this work, we leverage a statistical perspective on providing an unbiased estimation of the model accuracy with the smallest estimation variance, named Stratified random Sampling with Optimum Allocation (SSOA).Methods: Our approach first divides the unlabeled test set into strata based on predictive confidences. Then, we design two stratum accuracy variance estimation methods to allocate the given budget assigned to each stratum based on the optimum allocation strategy. Finally, we conduct multiple experiments to evaluate the effectiveness and stability of SSOA by comparing it with baseline methods.Results: The results show that SSOA significantly outperforms all compared approaches with average improvements over 26.14% in terms of Mean Squared Errors (MSE) of estimated accuracy. In addition, the MSE shows a steady downward trend as the budget grows.Conclusion: SSOA can assist testers in estimating the accuracy of DNNs, lowering labeling costs, and enhancing the efficiency of DNN testing.

Automated summary

In this paper, a statistical method called Stratified random Sampling with Optimum Allocation (SSOA) is proposed to provide an unbiased estimation of model accuracy with the smallest estimation variance. The unlabeled test set is first divided into strata based on predictive confidences. Then, two stratum accuracy variance estimation methods are designed to allocate the given budget to each stratum based on the optimum allocation strategy. Multiple experiments are conducted to evaluate the effectiveness and stability of SSOA by comparing it with baseline methods.