Intra prediction mode decision using adaptive selection of candidate modes in HEVC

Abstract

In this thesis, we present method for intra prediction mode decision for fast intra coding in High Efficiency Video Coding (HEVC). Proposed method has two steps. In step 1, we reduce the full rate distortion (RD) candidates from the rough mode decision (RMD). For step1, two parts are used internally. Part 1 is reduction of the full RD candidates based on the statistical observation. We observe that among 8 or 3 candidates for each 64x64, 32x32, 16x16 or 8x8, 4x4 Prediction Units (PU), first 3 or 2 candidates can be the final best mode at 90% probability. Therefore, we use 3 or 2 candidates for full RD calculation. Part 2 is the pro-cess that checks whether un-selected 5 or 1 candidate(s) are added as a full RD candidate or not. In this part, we use the using adaptive binary table (ABT) that is made by using the probability of final best mode based on the mode that has the smallest RMD cost. In step 2, full RD candidates from step 1 are used for full RD cost calculation. In this step, threshold is used for early termination of the full RD cost calculation. The threshold value is calculated from Bayes decision with the two Rayleigh distributions that is approximated from the two groups of actual data - one group is the Best RD cost group, the other is the NotBest RD cost group. During full RD cost calculation in RMD cost order, if current full RD cost is smaller than the threshold cost value, then, we terminate the full RD cost calculation process for remained candidates. Experimental results demonstrated that the proposed method outperforms the previous methods, especially about the Bjontegaard distortion-rate (BD-rate) loss compared with the average time saving.