Video spatio-temporal resolution enhancement using deep learning

Abstract

Because videos can be displayed at high resolution and high frame rate with the development of display technology, the demand for high-quality video is increasing. However, due to the several limitations of the camera, video is often obtained with poor quality such as low resolution and low frame rate. In addition, blurry videos are obtained when an object or a camera moves with a relatively long exposure time. For these low-quality videos, they can be improved by applying video enhancement techniques. In this paper, we propose several video enhancement techniques that can improve the quality of videos. Firstly, we propose a new frame rate up-conversion method that improves the frame rate of video by generating and inserting intermediate frames between successive frames. To estimate and compensate reliable object motions between successive frames, we propose a new triple-frame-based bi-directional motion estimation method that utilizes three frames. In addition, we propose a new motion vector refinement method based on deep learning, which results in producing more elaborate intermediate frames. Secondly, we introduce a new video deblurring method. In our approach, both deconvolution and aggregation techniques are used for video deblurring. In the deconvolution part, blurry inputs are first preprocessed by non-local operations. Then, the preprocessed frame is aligned with adjacent preprocessed frames and deblurred by a deconvolution network. The multiple deblurred frames from the deconvolution network are combined in a latent frame according to reliability maps produced by an aggregation network. Lastly, we propose a new joint video enhancement method that generates high-resolution and high-frame-rate clear videos from low-resolution and low-frame-rate blurry videos. The videos that contain multiple degradation factors can be improved by applying video enhancement techniques sequentially. However, this cascade manner is sub-optimal and inefficient with respect to the processing time and network size. Our joint video enhancement method can effectively enhance the quality of videos because it solves a unified degradation model. Experimental results showed that the proposed video enhancement techniques outperformed other existing video enhancement methods in both subjective and objective evaluations.