High quality shape capture from an RGB-D image under uncalibrated natural illumination

Abstract

We present an optimization-based framework to estimate both natural lighting conditions and high quality shape information using a single RGB-D image of diffuse objects. To estimate accurate natural lighting conditions, we present a general lighting model consisting of global and local models. The global lighting model is robustly estimated from the RGB-D input with a low-dimensional characteristic of the diffuse lighting model. The local lighting model can represent spatially varying illumination due to attached shadows, inter-reflections, and near lightings. With both the global and local lighting model, we can model complex lighting variations that previous methods cannot account for. For high quality shape capture, a shape from shading approach is applied with the estimated lighting model. Use of a geometric normal constraint greatly reduces local ambiguity in determining local surface orientation. Since both lighting conditions and shape estimations are done with a single RGB-D image, our method can capture the high quality shape of dynamic objects under uncalibrated varying illumination conditions. Experimental results using RGB-D images of a variety of diffuse objects in natural lighting conditions demonstrate the feasibility and effectiveness of the method to dramatically improve the limited low depth resolution of depth cameras, such as Kinect.