NeRFFaceSpeech: one-shot audio-diven 3D talking head synthesis via generative prior

Abstract

Audio-driven talking head generation is advancing from 2D to 3D content. Notably, recent advancements leveraging Neural Radiance Field (NeRF) are in the spotlight to synthesize 3D output but they need extensive paired audio-visual data for each identity, limiting their scalability. On the other hand, some studies have demonstrated that even with a single image, it is possible to generate convincing audio-driven talking head synthesis. Despite their promise, as observed, these techniques struggle to produce accurate 3D-aware results due to insufficient information on obscured regions of a single image. In this paper, we propose our novel pipeline, NeRFFaceSpeech, which enables us to bridge the trade-off between the number of images and 3D information fidelity. Using prior knowledge of generative models combined with NeRF, our method can craft a 3D-consistent facial feature space corresponding to a single image. Following this, our approach employs ray deformation to map the audio-correlated vertex dynamics from a parametric face model to the facial feature space, ensuring realistic 3D facial motion. Moreover, to replenish the lacking information in the inner-mouth area, which can not be obtained from a given single image, we introduce LipaintNet—a novel network trained in a self-supervised manner. Lastly, our comprehensive experiments demonstrate the superiority of our pipeline for producing enhanced 3D consistency in generating audio-driven talking heads from a single image compared to previous approaches.