Image-based rendering

Abstract

Image-based rendering is an approach to generate an image viewed with arbitrary camera parameters using scene information extracted from a set of reference images. The rendering speed of this approach depends only on image size instead of scene complexity, and complex hand-made geometric models are not required. Hence, image-based rendering is appropriate to generate realistic images in interactive applications. In this thesis, we propose a new approach to image-based rendering that uses a set of optical flows as scene information. We develop an analysis technique that produces matching chains from sequences of stereo matching and extracts optical flows from the matching chains without any special equipments such as range cameras. We also derive validity conditions of optical flows and show that the obtained flows satisfy those conditions. This means that we can generate visually correct images from the optical flows even with specular reflection and object occlusion. Since environment geometry is inferred from the optical flows, we are able to generate more accurate images with this additional geometric information. We accelerate the rendering procedure by processing optical flows in back-to-front order and culling flows using the filtering search technique. It is also possible to compose an image rendered from optical flows with an image generated by a conventional rendering technique through a simple Z-buffer algorithm. Since the rendering time is significantly influenced by the number of optical flows, we propose a compact representation of optical flows that reduces the number of optical flows while encoding the same scene information. In this new representation, an optical flow is constructed for a pixel interval in which pixels have the same intensity. We are able to render more than ten images per second by using the new representation.