Invisible emotion magnification algorithm (IEMA) for real-time micro-expression recognition with graph-based features

The interest in real-time micro-expression recognition has increased with the current trend in human-computer interaction applications. Presently, there are several approaches which include geometrical analysis for fast facial muscle movement feature representations. However, due to the subtleness of emotions, recognition accuracy remains a challenge in this field. In order to address this challenge, this paper put forward a novel geometrical-based invisible emotion magnification algorithm (IEMA). This algorithm utilizes facial landmarks of on-set and apex frames to compute magnication elements as well as the direction of facial movement, and then applies them to the x and y coordinates of the apex-frame to enhance the magnitude of facial muscle movement for invisible emotion. Furthermore, this algorithm integrates a parameter which is set to adjust the magnification level in order to achieve optimal performance. The proposed IEMA was analyzed using a landmark-based facial graph, and subsequently, the euclidean distance and gradient of the graph segments were presented as features. The experimental results suggest that the proposed IEMA yields the highest accuracy of 94.78% and an F1-score of 0.90 with the CASMEII dataset, while the SMIC, CAS(ME)(2) and SAMM datasets yield accuracies of 87.98%, 88.89% and 94.72% with F1-score of 0.86, 0.83 and 0.88, respectively. In fact, the proposed IEMA outperformed several micro-expression magnification methods in terms of accuracy as well as the F1-score. In addition, the proposed IEMA running time is 3.41ms per sample, which translates to 293 fps. This suggests that the proposed IEMA processes samples above the minimum requirement of 200fps for micro-expression recognition in real-time.

Automated summary

This paper proposes a novel geometrical-based invisible emotion magnification algorithm (IEMA) for real-time micro-expression recognition. The algorithm utilizes facial landmarks to compute magnification elements and facial movement direction, and applies them to the coordinates of facial apex frames to enhance the magnitude of facial muscle movement. The experimental results show that the proposed IEMA outperforms other micro-expression magnification methods in terms of accuracy and F1-score, and meets the minimum requirement for real-time micro-expression recognition.