Title:Signal search enhanced RMT estimators for estimating the number of signals based on random matrix theory

Abstract:Estimating the number of signals is a fundamental problem in many scientific and engineering fields. As a well-known estimator based on the random matrix theory, the RMT estimator estimates the number of signals via detecting the largest noise eigenvalues. Since the RMT estimator does not consider the interaction among the eigenvalues when the number of samples is limited, the RMT estimator may down-estimate the number of signals when there are low signal-to-noise ratio (SNR) signals. In order to overcome this problem, in this paper we propose two signal search enhanced RMT estimators by considering the interaction among the sample eigenvalues. The main ideas of the proposed two signal search enhanced RMT estimators are as follows: (a) Firstly, we derive a novel decision statistics for signal number estimation by utilizing the asymptotic distribution of the sample signal eigenvalues, and derive the distribution of this decision statistics. Then, a signal search enhanced RMT estimator is proposed to estimate the number of signals via sequentially testing whether the decision statistics is above a threshold; (b) Secondly, we analyze the effect of the interaction among eigenvalues on the detection performance of the RMT estimator and the proposed signal search enhanced RMT estimator. Based on this analysis, a hybrid RMT is proposed by making use of the advantages of these two estimators while avoiding their individual drawbacks. Finally, simulation results show that the proposed two signal search enhanced RMT estimators outperforms the existing estimators including the RMT estimator, the classic AIC and MDL estimators, and the modified AIC estimator, especially for the cases when there are low SNR signals and the system size becomes larger.