Orthogonal forward selection and backward elimination algorithms for feature subset selection

Sequential forward selection (SFS) and sequential backward elimination (SBE) are two commonly used search methods in feature subset selection. In the present study, we derive an orthogonal forward selection (OFS) and an orthogonal backward elimination (OBE) algorithms for feature subset selection by incorporating Gram-Schmidt and Givens orthogonal transforms into forward selection and backward elimination procedures, respectively. The basic idea of the orthogonal feature subset selection algorithms is to find an orthogonal space in which to express features and to perform feature subset selection. After selection, the physically meaningless features in the orthogonal space are linked back to the same number of input variables in the original measurement space. The strength of employing orthogonal transforms is that features are decorrelated in the orthogonal space, hence individual features can be evaluated and selected independently. The effectiveness of our algorithms to deal with real world problems is finally demonstrated.