Evaluating Multimodal Wearable Sensors for Quantifying Affective States and Depression With Neural Networks

With the increasing proliferation of embedded sensors in wearable devices, there is potential for modeling individual emotional and mental state variations. The popular measure for the quantification of emotions outlines the affective states of arousal and valences, with high and low being the discrete categories of interest. Recent works explore the discernability of digital behavior differences between groups with and without mental disorders. However, the interaction between physiological states and affective states within a predominantly depressive population remains to be studied with the aid of wearables. Despite the pervasiveness of emotional state inference through the tracking of ubiquitous physiological trackers, such as heart rate, blood volume pulse, skin conductance, and motion, a dearth of work is noted in the exploration of physiological markers in single-modal and multimodal settings. This work provides an extensive evaluation of a convolutional neural network with an attention mechanism ensembled with a random forest algorithm to effectively leverage multiple raw signal-to-image transformations as feature inputs to predict depression severity and affective state. The proposed models are assessed on the Daily Ambulatory Psychological and Physiological recording for Emotion Research (DAPPER) dataset and achieve the sensitivity: specificity scores of 58.75%:45.59%, 62.34%:43.41%, and 49.43%:51.70% for predicting depression, valence, and arousal with a mixture of unimodality and bimodality applying continuous wavelet transforms and short-time Fourier transform to motion and skin-conductance readings, respectively. This work is envisioned as a preliminary study to contribute toward the monitoring of affective states among a depressed population by utilizing low-frequency sensor recordings with the DAPPER dataset.

Automated summary

With the increasing use of wearable devices with embedded sensors, there is potential for modeling individual emotional and mental state variations. While there have been studies exploring digital behavior differences between groups with and without mental disorders, the interaction between physiological states and affective states within a predominantly depressive population remains to be studied. This study proposes models that leverage multiple raw signal-to-image transformations to predict depression severity and affective state, and evaluates them using a dataset.