3D scene understanding from deep point cloud

Abstract

In modern times, the application of 3D scene comprehension spans a broad range of industries. A recent example of this includes Apple's release of VisionPro, a device dedicated to mixed reality, equipped with multiple camera sensors, depth sensors, and LiDAR sensors. Given this setup, it can be inferred that the device will process 3D data directly for 3D scene understanding. Beyond its application in mixed reality, autonomous vehicles and mobile devices also incorporate 3D scene comprehension. In light of this trend, this dissertation explores 3D Scene Understanding through Deep Point Cloud. This dissertation expands the definition of point cloud, involving three key aspects: (1) Point-from-Sensor, (2) Point-from-Model, and (3) Point-from-Surface samples. In terms of point-from-sensor, the traditional meaning of point cloud was limited to measurements taken from LiDAR sensors. However, we broaden this definition to also encompass measurements obtained from depth sensors. For point-from-model, contemporary deep learning methods employ readily available pre-trained networks to leverage geometric priors. Consequently, the definition of point cloud also incorporates the geometric information inferred from these pre-trained deep learning models. Lastly, point cloud includes surface samples. Analogous to Poisson sampling, points can be sampled in close proximity to the learned surface geometry within the Signed Distance Function. In this context, we consider this point as an element of the newly defined point cloud. Informed by this definition, this thesis delves into three typical 3D scene interpretation tasks: 3D reconstruction, 3D recognition, and neural rendering, approached from the point cloud perspective.

In the 3D reconstruction task, we directly confront the inherent challenges in point cloud, such as sparsity, noise, and irregularity. (1) It is widely acknowledged that point clouds from LiDAR sensors are sparse, which hinders the geometric understanding of target scenes. We propose stereo-LiDAR fusion methods to address this issue, which leverage dense stereo images with sparse points from a LiDAR sensor. (2) In regard to another aspect of point cloud issues, noise, we utilize multi-view stereo matching and harness multi-view cues for a point cloud denoising algorithm. (3) To tackle irregularity in point cloud, we utilize a novel sparse tensor representation for point cloud reconstruction.

The 3D recognition task, unlike the 3D reconstruction task, aims for semantic understanding, such as categories. According to the problem setup, instead of explicitly correcting or modifying point information, we suggest a method to mitigate the irregularity issue of point cloud. Specifically, we reevaluate the k-Nearest Neighbor algorithm so that our method treats the k-nearest neighbor clustering as a bi-directional graph, while most prior studies only use this clustering as a single-direction graph.

In the neural rendering task, we concentrate on enhancing the strengths of point cloud, such as efficiency and representational power. We propose a space-time surface sampling idea for high-fidelity rendering.

By employing this strategy, our goal is to highlight the importance of point cloud for 3D scene interpretation in real-world applications.