The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games

What if emotion could be captured in a general and subject-agnostic fashion? Is it possible, for instance, to design general-purpose representations that detect affect solely from the pixels and audio of a human-computer interaction video? In this article we address the above questions by evaluating the capacity of deep learned representations to predict affect by relying only on audiovisual information of videos. We assume that the pixels and audio of an interactive session embed the necessary information required to detect affect. We test our hypothesis in the domain of digital games and evaluate the degree to which deep classifiers and deep preference learning algorithms can learn to predict the arousal of players based only on the video footage of their gameplay. Our results from four dissimilar games suggest that general-purpose representations can be built across games as the arousal models obtain average accuracies as high as 85 percent using the challenging leave-one-video-out cross-validation scheme. The dissimilar audiovisual characteristics of the tested games showcase the strengths and limitations of the proposed method.

Automated summary

This article investigates the possibility of predicting emotions solely based on audiovisual information in videos using deep learned representations. The study shows that general-purpose representations can be built to detect emotions by relying only on the pixels and audio of a human-computer interaction video, as demonstrated in the domain of digital games.