Why did AI get this one wrong? - Tree-based explanations of machine learning model predictions

Increasingly complex learning methods such as boosting, bagging and deep learning have made ML models more accurate, but harder to interpret and explain, culminating in black-box machine learning models. Model developers and users alike are often presented with a trade-off between performance and intelligibility, especially in high-stakes applications like medicine. In the present article we propose a novel methodological approach for generating explanations for the predictions of a generic machine learning model, given a specific instance for which the prediction has been made. The method, named AraucanaXAI, is based on surrogate, locally-fitted classification and regression trees that are used to provide post-hoc explanations of the prediction of a generic machine learning model. Advantages of the proposed XAI approach include superior fidelity to the original model, ability to deal with non-linear decision boundaries, and native support to both classification and regression problems. We provide a packaged, open-source implementation of the AraucanaXAI method and evaluate its behaviour in a number of different settings that are commonly encountered in medical applications of AI. These include potential disagreement between the model prediction and physician's expert opinion and low reliability of the prediction due to data scarcity.

Automated summary

The increasing complexity of machine learning models has led to the development of black-box models, which are accurate but difficult to interpret. This article introduces AraucanaXAI, a novel method that generates explanations for machine learning predictions. The method uses surrogate, locally-fitted classification and regression trees to provide post-hoc explanations with superior fidelity, ability to handle non-linear decision boundaries, and support for both classification and regression. The authors also provide an open-source implementation and evaluate its performance in medical AI applications, including cases of disagreement with expert opinions and limited data reliability.