Enhancing Segment-Based Speech Emotion Recognition by Iterative Self-Learning

Despite the widespread utilization of deep neural networks (DNNs) for speech emotion recognition (SER), they are severely restricted due to the paucity of labeled data for training. Recently, segment-based approaches for SER have been evolving, which train backbone networks on shorter segments instead of whole utterances, and thus naturally augments training examples without additional resources. However, one core challenge remains for segment-based approaches: most emotional corpora do not provide ground-truth labels at the segment level. To supervisely train a segment-based emotion model on such datasets, the most common way assigns each segment the corresponding utterance's emotion label. However, this practice typically introduces noisy (incorrect) labels as emotional information is not uniformly distributed across the whole utterance. On the other hand, DNNs have been shown to easily over-fit a dataset when being trained with noisy labels. To this end, this work proposes a simple and effective iterative self-learning (ISL) framework, which comprises a procedure to progressively correct segment-level labels in an iterative learning manner. The ISL method produces dynamically-generated and soft emotion labels, leading to significant performance improvements. Experiments on three well-known emotional corpora demonstrate noticeable gains using the proposed method.

Automated summary

This paper presents a deep neural network approach for speech emotion recognition using a limited amount of labeled data. Unlike traditional methods, this approach trains backbone networks on shorter segments, thereby increasing the number of training examples. However, due to the lack of segment-level labels in most emotional corpora, an iterative self-learning framework is proposed to correct the labels and improve recognition performance.