Rate-distortion optimization-based intra encoder design

Abstract

In this paper, we proposed a rate-distortion (R-D) optimization-based intra encoder for better R-D performance. To reduce the computational complexity of R-D optimization-based mode decision, we proposed two schemes; an early block type decision method and a rate estimation method for context-based adaptive variable length coding (CAVLC). Using the first method, we decided early the best block size for R-D optimization-based mode decision among luma 4×4, 8×8 and 16×16 block. The proposed method is used for R-D cost of a subblock (8×16 block) instead of that of macroblock (MB) with the proposed encoding order for the intra prediction of 4×4 and 8×8 block. The luma components of total R-D cost are only used for the mode decision without chroma components. The chroma components have small potion of total R-D cost of MB due to subsampling (4:2:0) and relatively small variance between chroma pixels. This method achieved the reduction of computational complexity with average 0.04dB PSNR degradation and rate increase of less than 1% compared with the full search method. The second method is based on replacing the CAVLC table for three key parameters, $\textit{Coefficient\_token}$, $\textit{Total\_zeros}$, and $\textit{Run\_before}$ into simple arithmetic expressions. Experimental results show that the proposed scheme reduced the encoding time by about 50% for full high definition and achieves almost the same PSNR at about 0.1% bit-rate increment compared with the JM reference software.