Soft variable selection combining partial least squares and attention mechanism for multivariable calibration

Partial least squares (PLS) are a widely used algorithm for building a linear model between chemical properties and multivariables. Due to the abundant features and relatively few calibration samples, variable selection is usually adopted to eliminate uninformative variables and restrain overfitting. In this study, a new variable selection method, called 'Attention-PLS' was proposed, combining PLS with the attention mechanism in a neural network. The attention mechanism tries to find a new combination of the variables, and owing to the property of softmax function, only few variables' weights are dominant in the new combination's weights. Attention-PLS is a soft way of variable selection, as it does not absolutely eliminate the influence of the unimportant variables but enlarge their difference of variables' weights by using softmax function to normalize the weights. Attention-PLS is compared with some common methods like ordinary partial least squares, Least Absolute Shrinkage and Selection Operator (LASSO), Ridge Regression (RR), Monte Carlo based uninformative variable elimination (MC-UVE), and Sparse Partial Least Square (SPLS), which are applied to three near infrared spectral (NIR) datasets. The results show that the proposed method has better prediction performances.

Automated summary

In this study, a new variable selection method called 'Attention-PLS' was proposed, which combines PLS with the attention mechanism in a neural network to build a linear model between chemical properties and multivariables. The results show that Attention-PLS has better prediction performances.