CABAC rate estimation for H.264/AVC mode decision

Abstract

In H.264/AVC coders, optimal mode decision for each coding unit can be achieved based on the estimation of the rate-distortion (R-D) cost for each candidate mode. The rate-distortion cost considers both the bit rate and the video quality degradation of all possible modes. In this thesis, I focus on the bit rate estimation scheme that reduces the computational complexity of the output rate estimation of CABAC. CABAC is chosen due to its compression ratio advantage over CAVLC. We first establish a rate model for CABAC using probabilistic behavior of the arithmetic coding engine. The complex calculation of the arithmetic coding engine is replaced by a table lookup for rate estimation. With some techniques simplifying the context selection algorithm applied, the proposed rate estimator reduces about 30% of the computational complexity of the R-D optimized mode decision. The entire encoder is thereby accelerated by 17 $\sim$ 25% with almost no degradation in the R-D performance. For further speed improvements, we modify the rate estimator to implement it as hardware. The speed of the hardware depends only on the number of contexts used in a coding unit while the speed of conventional CABAC hardware implementations depends on the number of bins provided as input. Experimental results show that the proposed rate estimator can accelerate the estimation hardware by about 5 to 18 times, depending on the parameter QP and video sequence.