Spatial-Temporal Attention Network for Depression Recognition from facial videos

Recent studies focus on the utilization of deep learning approaches to recognize depression from facial videos. However, these approaches have been hindered by their limited performance, which can be attributed to the inadequate consideration of global spatial-temporal relationships in significant local regions within faces. In this paper, we propose Spatial-Temporal Attention Depression Recognition Network (STA-DRN) for depression recognition to enhance feature extraction and increase the relevance of depression recognition by capturing the global and local spatial-temporal information. Our proposed approach includes a novel Spatial-Temporal Attention (STA) mechanism, which generates spatial and temporal attention vectors to capture the global and local spatial-temporal relationships of features. To the best of our knowledge, this is the first attempt to incorporate pixel-wise STA mechanisms for depression recognition based on 3D video analysis. Additionally, we propose an attention vector-wise fusion strategy in the STA module, which combines information from both spatial and temporal domains. We then design the STA-DRN by stacking STA modules ResNet-style. The experimental results on AVEC 2013 and AVEC 2014 show that our method achieves competitive performance, with mean absolute error/root mean square error (MAE/RMSE) scores of 6.15/7.98 and 6.00/7.75, respectively. Moreover, visualization analysis demonstrates that the STA-DRN responds significantly in specific locations related to depression. The code is available at: https://github.com/divertingPan/STA-DRN.

Automated summary

This paper proposes a novel Spatial-Temporal Attention Depression Recognition Network (STA-DRN) that enhances feature extraction and relevance of depression recognition by capturing global and local spatial-temporal information. The experimental results demonstrate competitive performance and visualization analysis shows significant responses in specific locations related to depression.