(A) study on interactive 3D reconstruction using RGB-D camera

Abstract

3D object reconstruction is becoming an essential technology in various computer vision applications such as AR / VR and 3D printing. Recently, as commodity RGB-D cameras have been widely used, there is an increasing demand for a technique to acquire 3D model easily by using these sensors. Interactive 3D reconstruction is one of the most user-friendly reconstruction methods using RGB-D camera. For interactive 3D reconstruction, it is necessary to obtain consecutive depth maps with user interaction. If the consecutive depth maps (2.5D) of an object are input, the 3D model of the object can be reconstructed by fusing them into a global volume model based on their poses. There are two main problems in the interactive 3D object reconstruction: i) the complexity of object shape may not be sufficient to reconstruct an object, and ii) occluding depths caused by interaction can be integrated into the volume model. In this dissertation, we propose a robust interactive RGB-D object reconstruction system via fusion of multi-modal data (color and depth images) to overcome the aforementioned problems. Object information that can be obtained from a color image can be categorized into two main types as follows: silhouette, which represents the object region in the input image, and texture, which represents the inside information of the object region. First, we perform a pose estimation using both texture and depth information of an object to overcome pose drift caused by low shape variation of an object. In order to effectively use texture information, we propose an adaptive feature selection algorithm based on gradient variation. Second, we generate multi-modal visual hull shield by fusing silhouette and depth data of an object to remove dynamic occlusions caused by user interactions. To this end, we propose dual ray carving method for fusing data of different characteristics into a single object model.