Motion estimation (ME) is one of the most time consuming parts in video coding standard. As fast integer-pixel ME algorithm becoming more and more powerful, it is important to develop fast sub-pixel ME algorithm since the computational complexity of sub-pixel ME compared to integer-pixel ME has become relatively significant. In this paper, a novel fast sub-pixel ME algorithm is proposed. This algorithm first approximates the error surface of the sub-pixel position by a second order function and predicts the minimum point by minimizing the function at half-pixel accuracy. Then another second order approximation within a smaller area which is determined by the previous step is modeled to predict the best sub-pixel position. Experimental results show that the proposed method can reduce the sub-pixel search points significantly with negligible quality degradation.