Journal
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION
Volume 26, Issue 6, Pages 1281-1292Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TEVC.2022.3208110
Keywords
Bayesian inference; benchmarking; evolutionary algorithms; probabilistic models on permutation spaces
Funding
- Spanish Ministry of Science and Innovation [PID2019-104933GB-10/AEI/10.13039/501100011033, PID2019-106453GAI00/AEI]
- Basque Government [KK.2020/00049, KK.2021/00091, IT1504-22]
Ask authors/readers for more resources
This work delves into the Bayesian statistical assessment of experimental results, proposing a framework for analyzing multiple algorithms on multiple problems/instances by transforming experimental results into rankings and estimating the posterior distribution of the parameters of probability models. Various inferences regarding algorithm rankings are examined, and a Python package and source code implementation are provided for other researchers to utilize.
In the field of optimization and machine learning, the statistical assessment of results has played a key role in conducting algorithmic performance comparisons. Classically, null hypothesis statistical tests have been used. However, recently, alternatives based on Bayesian statistics have shown great potential in complex scenarios, especially when quantifying the uncertainty in the comparison. In this work, we delve deep into the Bayesian statistical assessment of experimental results by proposing a framework for the analysis of several algorithms on several problems/instances. To this end, experimental results are transformed to their corresponding rankings of algorithms, assuming that these rankings have been generated by a probability distribution (defined on permutation spaces). From the set of rankings, we estimate the posterior distribution of the parameters of the studied probability models, and several inferences concerning the analysis of the results are examined. Particularly, we study questions related to the probability of having one algorithm in the first position of the ranking or the probability that two algorithms are in the same relative position in the ranking. Not limited to that, the assumptions, strengths, and weaknesses of the models in each case are studied. To help other researchers to make use of this kind of analysis, we provide a Python package and source code implementation at https://zenodo.org/record/6320599.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available