Optical tracking systems are widely adopted in surgical navigation. An optical tracking system is designed based on the principle of stereo vision with high-precision and low cost. This system uses optical infrared LEDs that are installed on the surgical instrument as markers and a near-infrared filter is added in front of the Bumblebee2 stereo camera lens to eliminate the interference of ambient light. The algorithm based on the region growing method is designed and used for the marker’s pixel coordinates’ extraction. In this algorithm, the singular points are eliminated and the gray centroid method is applied to solve the pixel coordinate of the marker’s center. Then, the marker’s matching algorithm is adopted and three-dimensional coordinates’ reconstruction is applied to derive the coordinates of the surgical instrument tip in the world coordinate system. In the simulation, the stability, accuracy, rotation tests, and the tracking angle and area range were carried out for a typical surgical instrument and the miniature surgical instrument. The simulation results show that the proposed optical tracking system has high accuracy and stability. It can meet the requirements of surgical navigation.