Deep learning approaches for light-field 3D reconstruction

Abstract

Sophisticated 3D reconstruction are required, depending on the development of 3D printers. The most common method for 3D modeling is CAD(Computer-aided-design). However, it requires professional training. The most widely used method for 3D reconstruction in computer vision is a geometric method which uses multiple images from different viewpoints, and a photometric method of measuring the surface normal under different light directions. Geometric methods have difficulties in complicated computation process and obtaining fine-scale 3D structures, and photometric method produces accurate surface normal under limited capturing environments such as dark room. Therefore, we achieve a sophisticated 3D reconstruction while overcome limitations of previous works. In this dissertation, we present a new method for 3D reconstruction of a light field camera image using depth neural network: In order to obtain more detailed 3D depth information, we super-resolved the spatial and angular resolution of the light field images, the surface normal of scenes obtained through the deep neural network using only the shading information of one image. We also proposed a network restoring 3D reconstruction from the light field images. Finally, we combine the surface normal and the depth obtained by deep neural network to recover a 3D model in which a more detailed surface is expressed. First, we present the super-resolution method of light-field camera. The light-field camera developed at Stanford university consists of several arrangements of over 100 CMOS based cameras, and commercial light-field cameras, an array of microlens placed between lenses and sensors, so that direction of light rays are recorded. Previous work which computes a disparity from slope in an EPI (Epipolar Plane Image) of light-field then conduct super-resolution using it. However, it is difficult to apply to the real-world because it assumes ideal light-field images. On the other hand, we present first time a deep neural network which learns the relation of light-field image for super-resolution. In order to train various disparities, we also built about 200 light-field image datasets from different places. The trained network models and dataset are available online so that related researchers can easily access. Second, we present a deep learning network that can obtain surface normal of an object from a single image. In the conventional photometric methods, images are taken in a dark room where the direction of light is easy to compute in order to light calibration. In the proposed method, a sophisticated surface normal is obtained from a NIR (near infrared ray) image without light calibration by training a deep neural network. Visible light images provide redundant information such as color, while NIR images filter the visible light bandwidth to obtain a precise surface image, and are easy to capture because they are not affected by fluorescent light. In order to improve the accuracy of the surface normal, we combine angle loss, inerrability constraint and intensity loss which are widely used objective functions in existing photometric approach. In order to evaluate the versatility of the trained neural network, the network is evaluated by images taken in an trained environment. We also show that the trained network is not applied only to a specific camera, and fine-scale 3D information can be obtained from the deep neural network trained by NIR images using a gray scale image. Third, we address a deep neural network for restoration 3D model from a single light-field image. The most efficient method to estimate disparity map from a light-field image is using the slope of the EPI. However, the inherent camera structure of existing commercial light field cameras has a smaller angular resolution available due to the trade-off between spatial and angular resolution and also light-field images suffers from noise. To overcome the noise issue, multi-view stereo methods are adopted, it finds correspondences by matching photoconsistency in local patches between sub-aperture images. It provides a high accuracy disparity maps but requires a lot of computation due to the matching processing. The proposed network shows that fast computation and also comparable performance to the conventional methods. Finally, we combined disparity maps and surface normal results to create a 3D model in which the details of surface are represented. The 3D model obtained by the proposed method shows very fast computation speed and promising performance in various environments.