4.7 Article

Audio-driven talking face generation with diverse yet realistic facial animations

Journal

PATTERN RECOGNITION
Volume 144, Issue -, Pages -

Publisher

ELSEVIER SCI LTD
DOI: 10.1016/j.patcog.2023.109865

Keywords

Audio-driven talking face generation; Face; Face animation; Audio-to-visual mapping; Image synthesis

Ask authors/readers for more resources

This paper introduces a novel method called DIRFA, which can generate diverse and realistic facial animations for talking faces from the same driving audio. By designing a probabilistic mapping network, the audio signals can be autoregressively converted into a facial animation sequence, and a temporally-biased mask is introduced to model the temporal dependency of facial animations. Realistic talking faces can be synthesized using the generated facial animation sequence and a source image.
Audio-driven talking face generation, which aims to synthesize talking faces with realistic facial animations (including accurate lip movements, vivid facial expression details and natural head poses) corresponding to the audio, has achieved rapid progress in recent years. However, most existing work focuses on generating lip movements only without handling the closely correlated facial expressions, which degrades the realism of the generated faces greatly. This paper presents DIRFA, a novel method that can generate talking faces with diverse yet realistic facial animations from the same driving audio. To accommodate fair variation of plausible facial animations for the same audio, we design a transformer-based probabilistic mapping network that can model the variational facial animation distribution conditioned upon the input audio and autoregressively convert the audio signals into a facial animation sequence. In addition, we introduce a temporally-biased mask into the mapping network, which allows to model the temporal dependency of facial animations and produce temporally smooth facial animation sequence. With the generated facial animation sequence and a source image, photo-realistic talking faces can be synthesized with a generic generation network. Extensive experiments show that DIRFA can generate talking faces with realistic facial animations effectively.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.7
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available