Data-driven learning fatigue detection system: A multimodal fusion approach of ECG (electrocardiogram) and video signals

Fatigue could lead to low efficiency and even serious disaster. In the educational field, detecting fatigue could help adjust teaching strategies accordingly when a student is inactive, which can potentially improve learning efficiency. Despite numerous studies in fatigue detection, there is still a lack of multiple classifier systems capable of detecting fatigue in daily life (without specific stimulations). To initially alleviate this problem, this study develops a learning fatigue detection system using a multimodal approach with ECG and video signals, classi-fying a learner's state into three categories: alert, normal, and fatigued. To validate performance, the proposed system is tested on (i) an open-source dataset DROZY (n = 35) and (ii) a self-collected dataset captured in a learning environment (n = 92). The experimental results based on 10-fold cross-validation demonstrate that the system outperforms the state-of-the-art approaches, achieving a detection accuracy of 99.6% and 91.8% on the two datasets, respectively.

Automated summary

Fatigue can lead to low efficiency and accidents. Detecting fatigue in the field of education can improve learning efficiency. This study develops a multimodal learning fatigue detection system using ECG and video signals to classify a learner's state into alert, normal, and fatigued. Experimental results show that the system outperforms other methods, achieving detection accuracies of 99.6% and 91.8% on two datasets.