Deep learning approaches for video frame interpolation and super resolution

Abstract

Recently, various social networking services (SNS) and video platforms have gathered much video data. There have been reports claiming approximately 400 hours worth of videos being uploaded to the web every minute, that is leading to an estimation that videos will take over 82% of the Internet traffic by 2022. Due to this phenomenon, many companies are focusing on video related applications and services. In this era of abundant video data, unsupervised learning from videos are gathering much attention as well. We propose the following research on video frame interpolation for stabilization and super resolution. First, we introduce full-frame video stabilization via iterative frame interpolation. Video stabilization is a fundamental and important technique for higher quality videos. Prior works have extensively explored video stabilization, but most of them involve cropping of the frame boundaries and introduce moderate levels of distortion. We present a novel deep approach to video stabilization which can generate video frames without cropping and low distortion. The proposed framework utilizes frame interpolation techniques to generate in-between frames, leading to reduced inter-frame jitter. Once applied in an iterative fashion, the stabilization effect becomes stronger. A major advantage is that our framework is end-to-end trainable in a self-supervised manner. We show the advantages of our method through quantitative and qualitative evaluations comparing to the state-of-the-art methods. Second, we propose a real-time video stabilization method via self-supervised learning. Unlike the majority of methods that run offline, our approach is designed to run in real-time. Our framework consists of a rigid transformation estimation between given frames for global stability adjustments, followed by scene parallax reduction via smoothed flow for further stability. Then, an inpainting module fills in the missing margin regions created during stabilization, in order to reduce the amount of margin cropping as post-processing. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-the-art methods as well as offline methods that require camera trajectory optimization. Third, we propose a joint super resolution (SR) and frame interpolation algorithm which can be applied to video reconstruction and compression. Previous works have addressed super resolution and frame interpolation as separate topics. Although significant advances have been made for each field of research, the joint task has not been explored extensively. A naive solution of applying SR and frame interpolation (and in reverse order) is sub-optimal. We propose a joint SR and frame interpolation model via permutation invariance. Our approach demonstrates favorable results to sequential application of state-of-the-art SR and frame interpolation methods. Lastly, we conduct research on solving the limitations of frame interpolation. The remaining challenges for frame interpolation are fast object motion and nonlinear motion. In this work, we address the nonlinear motion via modeling with three input frames instead of the typical two frame input. As for fast object motion, we take a data-driven approach by augmenting data samples on-the-fly via adding objects to the video frames conveying flying motion. In the appendix, we propose video summarization via natural language and the problem of composing a story out of multiple short video clips. These tasks use progressive learning and optimization techniques respectively.