Deep predictive video compression using mode-selective uni- and bi-directional predictions based on multi-frame hypothesis

Abstract

Recently, deep learning based researches in the field of image processing are being actively conducted. Likewise, there are recent methods based on both non-linear transform and motion estimation using deep neural networks for image and video compression. Deep learning-based image compression has shown significant performance improvement in terms of coding efficiency and subjective quality. However, less effort has been relatively done on video compression based on deep neural networks. In this study, we propose an end-to-end deep predictive video compression network, called DeepPVCnet, using mode-selective uni- and bi-directional predictions based on multi-frame hypothesis with a multi-scale structure and a temporal-context-adaptive entropy model as follows: First, we propose a structure that compresses the current frame using multiple reference frames, not using a single reference frame as in recent methods. The method based on a single reference frame has a limitation in improving the coding efficiency because the neighboring frame information is limitedly used for compression of the current frame. Second, learned from the lesson of the conventional video codec, we firstly incorporate a mode-selective framework into our DeepPVCnet with uni- and bi-directional predictive modes in a rate-distortion minimization sense. Since the recent methods used either uni-directional or bi-directional predictions for the current frame, the coding efficiency can be limited for video compression. Third, we propose an entropy model that utilizes the temporal context information of the reference frames for the current frame coding. The autoregressive entropy models for CNN-based image and video compression is difficult to compute with parallel processing. On the other hand, our proposed entropy model utilizes temporally coherent context from the reference frames, so that the context information can be computed in parallel. Finally, Our DeepPVCnet jointly compresses motion information and residual data that are generated from the multi-scale stucture via the feature transformation layers, which has an advantage in terms of computational complexity and the ability to remove redundancy between joint information. Extensive experiments show that our DeepPVCnet outperforms AVC/H.264, HEVC/H.265 and state-of-the-art methods in MS-SSIM perspective.