Statistical Stratification and Benchmarking of Robotic Grasping Performance

Robotic grasping is fundamental to many real-world applications, and new approaches must be systematically evaluated. However, in most cases, the performance of a specific approach is assessed by simply counting the number of successful attempts in a given task, and this success rate is then compared to those of other solutions, without taking into account the random variability across different experiments (e.g. due to sensor noise or variations in object placement). In order to address this issue, we classify the observed performance into qualitatively ordered outcomes, thereby stratifying the results. We then show how to analyze these results in a statistical framework, which accounts for the variability between experiments. The advantages of our approach are demonstrated in the practical comparison of four grasp planning algorithms. In particular, we show that the proposed approach allows us to carry out several distinct evaluations from a single set of experiments, without having to repeat the data collection process. We demonstrate that differences between the algorithms, which would not be apparent from overall success rates, can be identified and evaluated.

Automated summary

Robotic grasping is important for practical applications and needs systematic evaluation. However, current evaluation methods often ignore the random variability in different experiments. To address this, we propose a statistical framework to analyze the performance of grasp planning algorithms, considering the variability between experiments. Our approach allows for distinct evaluations without repeating data collection and enables identification and evaluation of differences between algorithms that may not be apparent from overall success rates.