Error assessment and optimal cross-validation approaches in machine learning applied to impurity diffusion

Machine learning models have been widely utilized in materials science to discover trends in existing data and then make predictions to generate large databases, providing powerful tools for accelerating materials discovery and design. However, there is a significant need to refine approaches both for developing the best models and assessing the uncertainty in their predictions. In this work, we evaluate the performance of Gaussian kernel ridge regression (GKRR) and Gaussian process regression (GPR) for modeling ab-initio predicted impurity diffusion activation energies, using a database with 15 pure metal hosts and 408 host-impurity pairs. We demonstrate the advantages of basing the feature selection on minimizing the Leave-Group-Out (LOG) cross-validation (CV) root mean squared error (RMSE) instead of the more commonly used random K-fold CV RMSE. For the best descriptor and hyperparameter sets, the LOG RMSE from the GKRR (GPR) model is only 0.148 eV (0.155 eV) and the corresponding 5-fold RMSE is 0.116 eV (0.129 eV), demonstrating the model can effectively predict diffusion activation energies. We also show that the ab-initio impurity migration barrier can be employed as a feature to increase the accuracy of the model significantly while still yielding a significant speedup in the ability to predict the activation energy of new systems. Finally, we define r as the magnitude of the ratio of the actual error (residual) in a left-out data point during CV to the predicted standard deviation for that same data point in the GPR model, and compare the distribution of r to a normal distribution. Deviations of r from a normal distribution can be used to quantify the accuracy of the machine learning error estimates, and our results generally show that the approach yields accurate, normally-distributed error estimates for this diffusion data set.