Face 3D reconstruction from single camera with dual pixel sensor

Abstract

Dual pixel (DP) sensors built with two photodiodes under a tiny microlens provide a narrow baseline stereo images. Recently, many mobile manufacturers have adopted DP in their flagship models because it supports the faster auto-focus and more aesthetic image captures. Despite of the advantages of DP, research on their use for facial recognition has been limited, due to the lack of a public dataset. In this paper, we present the first DP facial geometry dataset involving more than 100K face images, the corresponding full 3D models in metric scale, and normal maps for 107 subjects. To acquire the datasets, we design a new multi-camera array consisting of 8~DSLR cameras with DP sensors, and propose a Structured Light (SL)-based facial 3D reconstruction method. Moreover, we propose a new stereo matching network for dual pixel images, called SubMat. We design a subpixel-level matching module tailored for stereo matching of extremely narrow baseline images. Extensive experiments show SubMat enables to produce the accurate depth maps and the estimates help to detect face spoofing. We demonstrate that the proposed method trained on our facial datasets achieves state-of-the-art performance on DP stereo matching.