A Music-Driven Deep Generative Adversarial Model for Guzheng Playing Animation

To date relatively few efforts have been made on the automatic generation of musical instrument playing animations. This problem is challenging due to the intrinsically complex, temporal relationship between music and human motion as well as the lacking of high quality music-playing motion datasets. In this article, we propose a fully automatic, deep learning based framework to synthesize realistic upper body animations based on novel guzheng music input. Specifically, based on a recorded audiovisual motion capture dataset, we delicately design a generative adversarial network (GAN) based approach to capture the temporal relationship between the music and the human motion data. In this process, data augmentation is employed to improve the generalization of our approach to handle a variety of guzheng music inputs. Through extensive objective and subjective experiments, we show that our method can generate visually plausible guzheng-playing animations that are well synchronized with the input guzheng music, and it can significantly outperform the state-of-the-art methods. In addition, through an ablation study, we validate the contributions of the carefully-designed modules in our framework.

Automated summary

In this article, a fully automatic, deep learning based framework is proposed to synthesize realistic upper body animations based on novel guzheng music input. The proposed approach utilizes a generative adversarial network (GAN) to capture the temporal relationship between the music and the human motion data. Extensive experiments show that the method can generate visually plausible guzheng-playing animations that are well synchronized with the input guzheng music, outperforming the state-of-the-art methods. Ablation study validates the contributions of the carefully-designed modules in the framework.