Directional perception in man-made environments

Abstract

Man-made (structure) environments surrounding us have structural forms (from the layout of a city to buildings and many indoor objects such as furniture), which can be represented by a set of parallel and orthogonal planes. In 3D space, specifically, a major fraction of surfaces can be described by just a few planes with even fewer different surface normal directions. Also, intersections of planes in 3D are lines which can be observed as lines in the image space. A vanishing point (VP) is the intersection of multiple such image-space lines where the lines in 3D are all parallel to each other. This sparsity is evident in both the surface normal distributions and image-space lines. Thus, it naturally connects to structure assumptions in computer vision and 3D reconstruction systems. Most approaches focus on the segmentation or scene understanding task given known 3D structure, camera poses and intrinsically calibrated RGB images.

Among the various structure assumptions, Manhattan world (MW) assumption is commonly utilized due to its simplicity represented by three orthogonal directions and its notion can be represented as Manhattan frame (MF). Recently, more complex structure assumptions such as Atlanta world and mixture of Manhattan frames were proposed to represent more general man-made environments. Recognizing underlying structure assumptions of man-made environments (directional perception) is a key part in many computer vision applications such as indoor or urban 3D reconstruction, AR/VR where both accuracy and efficiency are required.

In this dissertation, under the man-made environments, given surface normals (3D domain) or line normals (intrinsically calibrated image domain), we propose a method to estimate directional perception (Manhattan frame and Atlanta frame) in a robust manner. In particular, based on a branch-and-bound (BnB) framework, which is robust against outliers and guarantees a global optimality, the contribution of this dissertation as follows: (1) We propose globally optimal MF estimation in real-time, which is applicable to real-time applications. Specifically, we present new bound computation to resolve computational time issue of conventional BnB framework. (2) Further, we propose new modeling for more general structural assumption, Atlanta world assumption and solve this problem in globally optimal manner. We introduce another efficient bound computation scheme and a method to estimate the number of horizontal direction for a given scene in an automatic manner. (3) Based on the estimated directional perception (especially, Atlanta frame), we introduce two interesting applications: AF-aware RGB-D SLAM and 3D object (vehicle) localization using AF.