Benchmarking Autonomous Scattering Experiments Illustrated on TAS

With the advancement of artificial intelligence and machine learning methods, autonomous approaches are recognized to have great potential for performing more efficient scattering experiments. In our view, it is crucial for such approaches to provide thorough evidence about respective performance improvements in order to increase acceptance within a scientific community. Therefore, we propose a benchmarking procedure designed as a cost-benefit analysis that is applicable to any scattering method sequentially collecting data during an experiment. For a given approach, the performance assessment is based on how much benefit, given a certain cost budget, it is able to acquire in predefined test cases. Different approaches thus get a chance for comparison and can make their advantages explicit and visible. Key components of the procedure, i.e., cost measures, benefit measures, and test cases, are made precise for the setting of three-axes spectrometry (TAS) as an illustration. Finally, we discuss neglected aspects and possible extensions for the TAS setting and comment on the procedure's applicability to other scattering methods. A Python implementation of the procedure to simplify its utilization by interested researchers from the field is also provided.

Automated summary

With the advancement of AI and machine learning, autonomous scattering experiments have great potential. To increase acceptance within the scientific community, it is crucial for these approaches to provide evidence of performance improvements. Therefore, a benchmarking procedure is proposed, which sequentially collects data during an experiment and applies cost-benefit analysis. Different methods can be compared and their advantages made explicit and visible. The cost measures, benefit measures, and test cases are defined for three-axes spectrometry (TAS) as an example.