Talking face generation by disentangled audio-visual representation and implicit natural code

Abstract

This paper addresses making a video where a speaker of a video is changed to another identity to look natural. One related research, called Disentangled Audio-Visual System(DAVS), got good results based on the way it disentangles audio-visual representation from images. However, DAVS generates static results that only move the mouth area. Based on DAVS, I aim to make dynamic results with head pose and facial expression. I approach this issue by adding implicit natural code. It is an expanded idea of Zhou et al., called Pose Controllable Audio-Visual System(PCAVS), that controls the pose of talking faces with only 12 dimensions vector, not by 2D or 3D landmark, to implicit natural code. In addition, I add a discriminator for accurate lip sync. With my model armed with the aforementioned components, I conduct comparative experiments with existing talking face generation models using a large-scale audio-visual dataset. In the case of videos with little movement, my model generates visual quality results equivalent to or better than other models. Also, ablation studies for a natural code and a discriminator for lip sync are conducted. It shows that each component does its job well.