Image retargeting and matting for new generation display

Abstract

As technology advances, new types of displays such as flexible screens or modular TV have emerged. Along with these hardware technologies, software must also be developed to fully utilize those displays. In this dissertation, I deal with content-aware image and video retargeting techniques that are essential in changing aspect ratios of the display, and matting, a key technology for movie and image editing. Content-aware image retargeting is a technique that preserves the main object in the image as much as possible when changing the aspect ratio of the image. Video matting is a technique for estimating the transparency of the foreground designated by the user. Image retargeting and matting are performed through visual scene understanding such as semantic information, temporal information, structure of foreground object and depth map in an image. The contributions of this dissertation are as follows. First, to maintain the aspect ratio of semantically important parts when performing image retargeting, we utilize high-level features based on deep learning. So far, retargeting methods have used low-level features, and recently, they have begun to use pre-trained deep-learning networks. This is because until now there has been no way to learn deep learning networks for retargeting due to the lack of datasets for retargeting. In this paper, we propose an end-to-end deep learning network and a weakly-supervised training method that uses only image-level annotation to perform retargeting. Owing to visual understanding via high-level features, the proposed method preserves the semantic area better than conventional methods. Second, extending single image retargeting to video retargeting is not straitforward. This is because it is difficult to simultaneously satisfy "content preservation", "spatio-temporal consistency" and "maintaining aspect ratio of the main object" in the video. In this dissertation, I recognize that the importance of the three elements mentioned above differs according to the characteristics of the video, and appropriately adjusts the three factors. I also propose a recurrent deep learning model to deal with temporal information and introduce a dataset construction method for video retargeting. Third, I propose a single image matting method based on deep learning. I observe that different image matting techniques work well depending on the structure of foreground object boundaries. Specifically, nonlocal based methods are good for long hair, and local based methods for solid boundaries. Therefore, this study suggests a deep learning based matting method that adaptively works according to foreground boundary structure. The proposed method takes initial alpha mattes and a RGB image obtained from the closed form matting and the KNN matting, and directly produces a high quality alpha matte. Furthermore, I propose "RGB guided JPEG artifact removal network" for compressed images in JPEG format. Finally, I provide image matting method on light-field image. I propose an algorithm that automatically generates a trimap using a depth map obtained through EPI analysis. Using the automatically generated trimaps and EPI correspondences, I introduce a method of estimating consistent alpha mattes across light-field images. I also create a light-field matting dataset for algorithm evaluation. Through this dissertation, I propose the retargeting and matting methods to fully utilize new generation displays. The proposed methods provide a new paradigm for image and video retargeting, image matting and light-field matting, and are expected to have a significant impact on following researches. In addition, the algorithms proposed in each method can be utilized for other application problems in the field of computer vision.