The H.264/AVC achieves higher coding performance than the previous video coding standards, such as MPEG-2 and MPEG-4. In order to achieve this improved coding performance, H.264 adopts various advanced techniques including multiple reference frame motion estimation (MRF-ME), which requires very heavy encoding complexity in proportion to the number of reference frames. In this letter, we propose an efficient priority-based reference frame selection (PRFS) method to reduce the computational complexity. The PRFS method computes the priorities of all reference frames using spatial and temporal correlation of the reference frame index and motion vectors, and selects candidate reference frames to reduce the number of reference frames for each ME block. Experimental results show that the proposed method reduces the computational complexity of the MRF-ME and achieves similar coding performance to that of previous methods.