Audiovisual speech recognition for Kannada language using feed forward neural network

Audiovisual speech recognition is one of the promising technologies in a noisy environment. In this work, we develop the database for Kannada Language and develop an AVSR system for the same. The proposed work is categorized into three main components: a. Audio mechanism. b. Visual speech mechanism. c. Integration of audio and visual mechanisms. In the audio model, MFCC is used to extract the features and a one-dimensional convolutional neural network is used for classification. In the visual module, Dlib is used to extract the features and long short-term memory recurrent neural network is used for classification. Finally, integration of audio and visual module is done using feed forward neural network. Audio speech recognition of Kannada dataset training accuracy achieved is 93.86 and 91.07% for testing data using seventy epochs. Visual speech recognition for Kannada dataset training accuracy is 77.57%, and testing accuracy is 75%. After integration, audiovisual speech recognition for Kannada dataset train accuracy is 93.33% and for testing is 92.26%.

Automated summary

This study develops an audiovisual speech recognition system for the Kannada language, utilizing the mechanisms of audio and visual speech as well as their integration. Through training and testing, the integrated system achieves a high level of accuracy.