Real-time lighting estimation for photorealistic augmented reality

Abstract

In augmented reality (AR), photorealistic rendering of virtual contents inserted into the physical world is one of the key factors for providing immersive experiences to users. Especially, estimating and applying a real-world lighting condition enable virtual objects to have realistic shading and shadow effects through the physical process of the light. As a result, users can have perception of realism and presence of the virtual objects. However, it is a challenging problem to estimate incident lights without using special gadgets such as a camera with fish-eye lens. Moreover, inferring dynamically changing illumination is highly difficult, which is often found in an AR environment. Although previous approaches from computer graphics and vision areas effectively solved an inverse rendering problem, including lighting estimation, using various computational algorithms, most of them required high computation costs, thus causing difficulties in their application to AR. To effectively handle such challenging problem, this dissertation proposes real-time methods for estimating a lighting condition in AR. To be specific, it contains progressively developing methods, including the deep learning-based approach to estimate omnidirectional incident lights by observing the reflected lights from a reference object, outdoor lighting estimation using global-scene data such as street-view images and 3D geometry, and more practical lighting estimation which is robust to the general input images captured in various indoor and outdoor scenes. Through these methods, this dissertation finds a more optimized and effective light-estimating method to enhance the visual coherence between the real world and the virtual objects, which finally allows users to have realistic AR experiences.