SeamsTalk: seamless talking face generation via flow-guided inpainting

Abstract

Talking face generation aims to generate a face video to speak according to a given audio or driving video. Despite the importance of natural lower face movement, previous approaches have focused only on animating the lip, neglecting the connection between the modified lower face and the original background. As a result, the generated face is not smoothly integrated into the original video. To address this, we propose a new method to create a seamless talking face video by reformulating talking face generation as a conditional video in-painting. Moreover, since previous methods solely rely on referencing the original frame, the original frame's lip shape influences a generated lip shape. Therefore, we devise a two-stage pipeline that leverages the original frame to reduce scene-specific information loss in the lower face and then utilizes multiple other frames to generate a desired lip shape. Experimental results demonstrate that our method generates a seamless talking face while maintaining similarly accurate lip shapes compared to existing methods.