This paper proposes a new binary motion estimation algorithm that improves the motion vector accuracy by using a hybrid distortion measure. Unlike conventional binary motion estimation algorithms, the proposed algorithm considers the sum of absolute difference (SAD) as well as the sum of bit-wise difference (SBD) as a block-matching criterion. In order to reduce the computational complexity and remove additional memory accesses, a new scheme is used for SAD calculation. This scheme uses 8-bit data of the lowest layer already moved into the local buffer to calculate the SAD of other higher binary layer. Experimental results show that the proposed algorithm finds more accurate motion vectors and removes the blockishness of the reconstructed video effectively. We applied this algorithm to existing video encoder and obtained noticeable visual quality enhancement.