Image generation by deep learning for augmented reality systems

Abstract

Augmented reality is a technique that enhances user experience by adding virtual images to the real world environment. In augmented reality, rendering virtual objects realistically is an important challenge for users to feel like the real world. Today, environmental mapping with light estimation is often used, but this method has some limitations. Estimating light sources is an ill-posed problem, and extracting accurate environment maps is difficult. Although the exact environment map is extracted, it is also difficult to generate realistic images with environment mapping compared to other physically-based rendering techniques. This dissertation presents a way to generate rendering results in the augmented reality system without the light estimation process using deep learning. Given background images and 3d modelings, features are extracted and combined. Then from the combined feature, our network creates images that render 3d models in the background. This method enables an expression of complex light interactions and the generation of realistic images compared to previous methods based on light estimation.