Depth and image sensing system with offset pixels: its hardware accelerator and focus-diversity system

Abstract

Depth sensing has been widely used in many applications over the past several decades. However, conventional depth sensing platforms are not suitable for mobile devices due to their large size and computation complexity. In this dissertation, we propose a compact system for depth and image sensing with offset pixel apertures (OPA). Specifically, we design an enhanced depth map processing (EDMP) and an OPA-dedicated image signal processing (OISP). The EDMP significantly improves the depth quality on excessively dark or light regions, greatly mitigates false depth aggregation at object boundaries, and increases depth precision with a very small overhead. Shot-noise-modeling and two-pass cost aggregation significantly reduce depth errors with reasonable hardware costs. Also, the OISP provides primary image processing while using white pixels to further improve color quality over conventional image sensors. Both EDMP and OISP are integrated on a single-chip. Our system shares the common hardware engine through time-multiplexing, taking advantage of the temporal margin caused by the sparsity of the OPA pixels. In addition, this dissertation proposes a novel focus-diversity camera system to extend the depth range. Since the narrow DoF(depth of field) of the existing single-focus system degrades the quality of depth information, the focus-diversity camera system can be configured to merge variously focused images to complement each other. The proposed method was implemented and verified as a mobile app to extract depth by commercial smart phone for practical use.