Audio-driven 3D visual dubbing with mask generation

Abstract

This paper discusses the method of generating a new 3D character dubbing animation using 3D character facial expressions and voice audio. The proposed approach aims to create an animation that preserves the facial expressions of the input animation while incorporating appropriate mouth movements for the voice audio. Deep learning is employed to extract expression information from the input facial animation and generate mouth movement information from the provided voice audio. Based on this information, a new animation is generated, and an additional mechanism is introduced to compare the expression information between the input facial animation and the output dubbed animation, thereby enhancing performance. Additionally, a masking process is trained to separate the regions surrounding the mouth of the 3D facial model from other areas, enabling the prediction of this division without manual intervention.