Stereo matching based on under- and over-segmentation, and occlusion handling

Abstract

The aim of stereo matching is to compute the depth information from which the geometric information of a scene can be obtained to reconstruct three dimensional structure of the scene. To do this, stereo matching needs solution to stereo correspondence problem, and finding an accurate solution to correspondence problem is the key to success of stereo matching and its application such as image- and video-based rendering, virtual view synthesis, $\it{etc}$. In this dissertation, stereo matching based on under- and over-segmentation is considered. In general, over-segmentation-based algorithms perform well on textured regions but require special provision to handle untextured regions, and the converse is true for under-segmentation-based algorithms. Unlike other over-segmentation-based algorithms, the proposed algorithm uses over-segmentation as well as under-segmentation to handle untextured regions. The disparity map is estimated based on over-segmentation, which is achieved by further segmenting the large segments of the under-segmented image. The over-segment disparity estimate in untextured regions is constrained to be close to the under-segment disparity estimate obtained by plane-fitting. This dissertation also considers a novel occlusion detection and handling algorithm using the disparity estimate of the match image. Incorporating energy terms that appropriately handle untextured regions as well as textured regions and occlusions, a global energy minimization framework is constructed based on a pairwise Markov random field, and the maximum a posteriori estimation using loopy belief propagation is performed to infer the disparities. Using both under- and over-segmentation, performance is improved over other over-segmentation-based algorithms, and the experimental results demonstrate the effectiveness of the proposed occlusion detection and handling algorithm.