The optimization of buffer gas pressure is very important to improve the performance of the rubidium (Rb) atomic magnetometer. In this paper we briefly introduce the basic principle and the experimental method of the rubidium magnetometer based on Faraday rotation effect, and describe the factors affecting the magnetometer sensitivity, then analyze and summarize the mechanism of the influence of spin-exchange, spin-destruction collisions, radiation trapping and the spin diffusion on spin relaxation of Rb atoms. Based on this, the relationship between the rubidium magnetometer sensitivity, the spin relaxation rate and the gas chamber conditions (buffer gas pressure, the bubble radius, measuring temperature) is established. Doing calculations by the simulation software, how the magnetometer sensitivity and the relaxation rate vary with the gas chamber conditions can be seen; finally, the optimal values of the buffer gas pressure under certain gas chamber conditions are obtained. The work is significant for the engineering development of rubidium magnetometer.