Generative emotional AI for speech emotion recognition: The case for synthetic emotional speech augmentation

Despite advances in deep learning, current state-of-the-art speech emotion recognition (SER) systems still have poor performance due to a lack of speech emotion datasets. This paper proposes augmenting SER systems with synthetic emotional speech generated by an end-to-end text-to-speech (TTS) system based on an extended Tacotron 2 architecture. The proposed TTS system includes encoders for speaker and emotion embeddings, a sequence-to-sequence text generator for creating Mel-spectrograms, and a WaveRNN to generate audio from the Mel-spectrograms. Extensive experiments show that the quality of the generated emotional speech can significantly improve SER performance on multiple datasets, as demonstrated by a higher mean opinion score (MOS) compared to the baseline. The generated samples were also effective at augmenting SER performance.(c) 2023 Elsevier Ltd. All rights reserved.

Automated summary

Despite the lack of speech emotion datasets, this paper suggests using synthetic emotional speech generated by an end-to-end text-to-speech (TTS) system to augment speech emotion recognition (SER) systems. The proposed TTS system includes encoders for speaker and emotion embeddings, a sequence-to-sequence text generator, and a WaveRNN for audio generation. Experimental results show that the generated emotional speech significantly improves SER performance on multiple datasets and effectively augments SER performance.