Robust 3D depth estimation for computational imaging

Abstract

We can easily observe image quality degradations from image noise or under-expose problems in low-light conditions due to inherent structural problems of conventional cameras. To solve these problems, this paper proposes computational imaging approaches that modify camera structures and design algorithmes for the cameras. In particular, we introduce multiple images acquisitions with different characteristics using the proposed camera structures and novel 3D depth estimation to match them. First, we propose an accurate depth estimation method based on light-field images. Unlike traditional cameras, light-field cameras sample the angular and spatial information of the incoming light on the sensor in a single photographic exposure. This produces well-aligned multi-view images, and allows refocusing by extending the depth field of the camera. However, inherent structural problems in light-field cameras result in a narrow baseline between the multi-view images, and severe image noise. Accurate depth maps were obtained from the light-field image using the proposed phase-shift and learning based multi-view stereo matching method. By applying the estimated depth map, we achieved high-quality denoised images. Second, we present a stereo matching process using monochrome and color cameras for accurate depth estimation and high-quality color images in low-light conditions. A unique difference between monochrome and color cameras is the presence or absence of a Bayer filter. Although the Bayer filter captures color information by filtering the light spectrum according to wavelength range, it occludes a lot of incoming light, which amplifies image noise in low-light conditions. On the other hand, a monochrome camera uses all the incoming light at each pixel and does not require a demosaic process. This gives them much better quantum efficiency and provides shaper images. We combined the advantages of these two cameras using the proposed iterative stereo matching approach. We recovered high-quality color images by transferring chrominance information about the color images into monochrome images using the estimated depth maps. Third, we introduced a novel camera array consisting of two color and two monochrome cameras with different ISOs or exposure times, which allowed a depth map and an exposure fusion image, practical alternative to an HDR image, to be obtained in a single photographic shot. We computed the depth map using images captured from our camera array. With the estimated depth map, we aligned the images to obtain an exposure fusion image. The algorithm are based on the depth from the light-field image and the stereo matching from the monochrome and color cameras provided robust performance in low-light conditions. In addition, we demonstrated that the proposed method enables the capture of exposure fusion video, using sequential images as input. However, there is still issue on a matching error from motion blur. Lastly, we present a coded exposure video for multi-image motion deblurring. In conventional photography, moving objects or moving cameras result in motion blur. The exposure time defines a temporal box filter that smears the moving object across the image by convolution. This box filter destroys important frequency components so that deblurring becomes a problem. The proposed solution is to capture video frames with a well-designed set of camera shutter fluttering patterns, which preserves the frequency components and makes deblurring tractable. Blurred images taken by the proposed method are deblurred by an iterative matting and deconvolution method.