In this paper, we introduce a flexible and simple system calibration method for specular surface metrology based on the combination of reflection rays determined by the varied points on a screen and reflection images of a plane mirror without fiducials placed at three different locations. This calibration procedure involves three steps. The camera is first calibrated based on plane patterns. Then the reflection ray directions are measured via correspondence matching. The last calibration step is the pose estimation by the orthogonal iteration algorithm and reflections in a plane mirror. Basically, the concept of replacing the coordinates of the camera center with the reflection ray can alleviate the trouble of imaging aberration. Then global optimization can be operated with the orthogonal projection defined by the reflection ray, providing precise initial values for the process of bundle adjustment, compared to the classical calibration approach directly using the local optimization algorithm. Simulations and experiments both demonstrate the validity, efficiency, and robustness of the proposed improved method. In the simulations, the proposed method achieves the absolute errors of the camera parameters within 3 pixels and the relative errors of the screen pose are below 0.5% when the noise level is 0.6 pixel. Furthermore, the calibration method shows strong anti-noise ability, relying on the application of the reflection rays and the global optimization before the final bundle adjustment. In addition, the reconstruction accuracy in our experiment improves by 60.11% by the proposed method compared with the calibration procedure, which only utilizes the bundle adjustment optimization. In general, this novel calibration method can make the measurement achieve high accuracy and robustness at a low cost and with a simple setup, providing an efficient, economical, and flexible approach for a phase measuring deflectometry system in practical situations.