Empirical analysis of deep learning networks for affective video tagging

This paper presents, a thorough evaluation of popular deep learning models to analyze and classify electroencephalogram (EEG) data for characterizing human affective states for video content tagging and retrieval. We use two pre-trained convolutional neural network (CNN) models AlexNet and GoogLeNet, and a Long Short Term Memory (LSTM) model to classify EEG data into appropriate affect categories using trans-domain learning. The purpose behind the use of pre-trained networks or trans-domain learning is twofold - to establish the versatility of pre-trained networks by testing their ability to classify EEG data for emotion recognition and the other is to reduce over cost of computation while training the networks. Our work tries to establish the answer of a simple question: Are pre-trained deep models versatile enough for classifying not only similar type of problems but are also effective for classifying problems pertaining to completely different domains? Also, using pre-trained models saves considerable computation time required for training a new model from scratch and fine tuning it. We use DEAP dataset for training and evaluation of these networks over a single modality 'valence' to simplify the comparison among these networks. Experiments are carried out by training the networks on EEG recordings obtained from single as well as multiple subjects to show the effects of subject-specific and generalized data on classification accuracy. Experimental results suggest the superiority of GoogLeNet for individual subject data while AlexNet outperforms other networks and has shown its capability of generalizing well. We compare the performance of these networks with state-of-art classifiers handcrafted by other authors for classifying EEG data and find that the performance of pre-trained CNNs used in our work are comparable or even better than the other handcrafted classifiers used by many authors.