External Validation and Prediction Employing the Predictive Squared Correlation Coefficient - Test Set Activity Mean vs Training Set Activity Mean

The external prediction capability of quantitative structure-activity relationship (QSAR) models is often quantified using the predictive squared correlation coefficient, q(2). This index relates the predictive residual sum of squares, PRESS, to the activity sum of squares, SS, without postprocessing of the model output, the latter of which is automatically done when calculating the conventional squared correlation coefficient, r(2). According to the current OECD guidelines, q(2) for external validation should be calculated with SS referring to the training set activity mean. Our present findings including a mathematical proof demonstrate that this approach yields a systematic overestimation of the prediction capability that is triggered by the difference between the training and test set activity means. Example calculations with three regression models and data sets taken from literature show further that for external test sets, q(2) based on the training set activity mean may become even larger than r(2). As a consequence, we suggest to always use the test set activity mean when quantifying the external prediction capability through q(2) and to revise the respective OECD guidance document accordingly. The discussion includes a comparison between r(2) and q(2) value ranges and the q(2) statistics for cross-validation.