Although the temporal scalability coding supported by both H.264/AVC and H.264/SVC provides a good means of adapting bitstream scalability in temporal dimension, it entails heavy computational complexity due to bidirectional motion estimation and compensation for B-pictures in temporal scalability levels. To make it more applicable, this paper introduces a fast inter-prediction mode decision method which is efficiently performed using statistical hypothesis testing. The hypothesis testing is performed on mean and variance of pixels in 16 x 16 and 8 x 8 blocks to decide early termination against further processing for their sub-blocks. Fallback check is employed to compromise the tradeoff between the compression efficiency and encoding time saving. The proposed method exhibits effective performance for early termination in the inter-prediction mode decision, thus leading to a significant reduction of the total encoding time up to average 64.8% with small increments in bit amounts and small degradation in visual quality.