(A) study on particle filtering based prediction for rates and distortions of video coding and an application to rate control

Abstract

(ii) less amounts of residues (texture information) due to improved motion prediction. So, it is a difficult problem to precisely estimate the texture bits occurred in various CU partition depths with various-sized block transforms. It is also challengeable to predict the non-texture bits such as motion, block partition and coding mode information etc. due to an increased number of various coding tools (modes) adopted for the improvement of coding efficiency. Therefore, it is worthwhile to estimate the total bits of texture and non-texture bits at the same time for each frame to be encoded. For this, an R-λ model has handled the total texture and non-texture bits at a time for RC and has worked reasonably well in HEVC. Nevertheless, if the rate estimation is inaccurately performed, that is, the R and λ values for a current frame cannot be linearly modeled with their respective values in the previous frames, the resulting RC performance is degraded. In our work, we adopt the RBE to precisely estimate the rates and then to allocate target bits based on the changes in the distortions of the previously coded frames, thus considering the rates and distortions simultaneously. Experimental results show that our proposed R/D estimation method significantly reduces the normalized root mean square error (NRMSE) from average 3.17 to 0.79 (74.90% reduction) for rate and from average 2.32 to 0.82 (64.61% reduction) for distortion, compared to the state-of-the art method. Furthermore, our RC method outperforms the RC of VTM-5.0 in terms of NRMSE with average 13.25% improvement, and maintains higher visual quality consistency in terms of standard deviation of PSNR by 23.31% improvement for All Intra (AI), average 27.38% and 9.10% for Low Delay (LD), and average 40.46% and 19.70% for Random Access (RA), respectively, compared to the default RC method of the original VTM-5.0.

An increased large number of block partition levels, enlarged transform kernel sizes and transforms of various kernel types adopted in the Versatile video coding TestModel (VTM) of MPEG and VCEG prohibit from precisely modeling the resulting various distributions of transform coefficient (TC) values, thus making the rate and distortion (R/D) estimation problems more challenging. The conventional R/D estimation methods first estimate the model parameters of a predefined probability density function (pdf) of TC values in Coding Tree Unit (CTU) partition depth levels or TC channels for the next frame. Then, R/D values are computed based on the estimated pdf for given quantization step sizes. This often leads to imprecise R/D estimation due to the mismatch between the true parameter of the pdf in the next frame and the parameter of the pdf estimated from the current and past frames. Furthermore, R/D estimation becomes more challenging for inter-predictive coding due to the lack of residues since prediction technologies have been greatly advanced. Rather, R/D estimation for intra-predictive coding is still capable with performing based on the modeling of TC distributions. In this dissertation, we propose a new R/D estimation method for video coding with deep block partitioning structures. In our proposed R/D prediction, we adopt a particle filtering based prediction (PFP) to precisely predict intermediate R/D estimates for the next frame in a stochastic manner, which helps increasing the prediction accuracy of fast changing R/D values. Then, based on the intermediate R/D estimates by PFP, we infer an optimal model parameter of the TC’s pdf via convex optimization. We found that the proposed method brings about more stable R/D estimation performance thanks to both the improved prediction accuracy using the PFP for abrupt changes in true R/D values and the precise estimation of the optimal model parameter. Based on our R/D estimation method by PFP, we extend our work to a frame-level constant bit-rate (CBR) control method using recursive Bayesian estimation (RBE) for Versatile Video Coding (VVC). The VVC adopts more complex coding structures with deeper depths and various block partitions (sizes and shapes) of Coding Unit (CU), compared to its former High Efficiency Video Coding (HEVC), thus leading to yield much improved coding efficiency. So, the conventional R-Q (rate-quantization) models such as a Laplacian mixture model (LMM) become inappropriate to be used for rate control (RC) due to two reasons: (i) the residues of various block-size transforms of CU partition blocks with deeper depths