A simulation of the impacts of machine learning to combine psychometric employee selection system predictors on performance prediction, adverse impact, and number of dropped predictors

We compare modern machine learning (MML) techniques to ordinary least squares (OLS) regression on out-of-sample (OOS) operational validity, adverse impact, and dropped predictor counts within a common selection scenario: the prediction of job performance from a battery of diverse psychometrically-validated tests. In total, scores from 1.2 billion validation study participants were simulated to describe outcomes across 31,752 combinations selection system design and scoring decisions. The most consistently valuable improvement from adopting MML over traditional regression was from dropping predictors rather than by improving prediction. On average, MML improved prediction of performance from psychometric scale composites only when the ratio of sample size to scale count was less than approximately 3, although algorithm choice, predictor count, and selection ratio affected outcomes as well. We also simulated the effects of design choices when combining item scores, which showed consistent, superior predictive accuracy for several MML algorithms, especially elastic net and random forest, over traditional regression. Given these results, we suggest the potential of machine learning for employee selection is unlikely to be realized in selection systems focusing on the combination of scale composites from previously validated psychometric tests. Instead, it will be realized in unconventional design scenarios, such as the use of individual items to make multiple trait inferences, or with novel data formats like text, image, audio, video, and behavioral traces. We therefore recommend researchers focus on the potential value of MML in future selection contexts rather than continuing to focus on the current value of MML in current selection contexts.

Automated summary

This study compares modern machine learning techniques with ordinary least squares regression in predicting job performance. The results show that the most valuable improvement from using machine learning is by dropping predictors rather than improving prediction accuracy. Machine learning only improves prediction accuracy when the sample size to predictor count ratio is less than approximately 3. The study also finds that several machine learning algorithms, especially elastic net and random forest, show superior predictive accuracy in combining item scores compared to traditional regression. Therefore, the potential value of machine learning is more likely to be realized in unconventional design scenarios and novel data formats.