The Necessity of Leave One Subject Out (LOSO) Cross Validation for EEG Disease Diagnosis

High variability between individual subjects and recording sessions is a known fact about scalp recorded EEG signal. While some do, the majority of the EEG based machine learning studies do not attempt to assess performance of algorithms across recording sessions or across subjects, instead studies use the whole data-set available for training and testing, using an established k-fold cross validation technique and thus missing performance in a real-life setting on an unseen subject. This study primarily aimed to show how important is to have a leave-one-subject-out (LOSO) evaluation done for any scalp recorded EEG based machine learning. This study also demonstrates effectiveness of a Multilayer Perceptron (MLP) in getting good LOSO accuracy from balanced, clean EEG data, without any pre-processing in comparison with traditional machine learning algorithms. The study used data from participants diagnosed with schizophrenia, as well as a group of participants with no known neurological disorder. Classification was done using traditional methods and MLP to classify the participants as belonging to disease or control subjects. Results shows that 85% accuracy on unseen subject was achievable from a clean data-set. MLP is seen to be effective in finding features by which schizophrenia could be detected from clean EEG data. LOSO evaluation done with this proven MLP configuration using carefully and intentionally corrupted data clearly indicate that for disease diagnosis, the k-fold classification result is misleading. Therefore, evaluation of any scalp recorded EEG based disease classification method must use a LOSO style cross-validation.

Automated summary

This study highlights the importance of leave-one-subject-out evaluation for machine learning based on scalp recorded EEG signals. It also shows the effectiveness of Multilayer Perceptron (MLP) in achieving high accuracy on unseen subjects from clean EEG data. LOSO evaluation with carefully corrupted data indicates that k-fold classification results may be misleading for disease diagnosis.