Title:Signal-noise search RMT estimator with adaptive decision criterion for estimating the number of signals based on random matrix theory

Abstract:Estimating the number of signals is a fundamental problem in many scientific and engineering fields. As a well-known estimator based on the random matrix theory (RMT), the RMT estimator estimates the number of signals via detecting the largest noise eigenvalues. Since the RMT estimator does not consider the interaction among eigenvalues, it tends to down-estimate the number of signals when there are weak signals. In order to overcome this problem, we propose signal search enhanced RMT estimators by considering the interaction among the sample eigenvalues. The main contributions are as follows: (a) We firstly derive a novel decision statistics for signal number estimation by utilizing the distribution of the sample signal eigenvalues. Then, based on this decision statistics, a signal search enhanced RMT estimator is proposed to estimate the number of signals via sequentially testing whether the decision statistics is above a threshold to determine whether an eigenvalue is from a signal or noise; (b) Secondly, the analysis shows that this signal search enhanced RMT estimator has better detection performance than the RMT estimator for weak signals, but has larger over-estimation probability than the RMT estimator for the cases when there is no signal or when all signals are strong signals. In order to overcome the individual drawbacks of these two RMT-based estimators, we further propose a signal-noise search RMT estimator whose decision criterion can be adaptively modified to avoid both the large over-estimation probability of the signal search enhanced RMT estimator and the large down-estimation probability of the RMT estimator. Finally, simulation results show that the proposed signal-noise search RMT estimator significantly outperforms the existing estimators including the RMT estimator, the classic AIC and MDL estimators, and the modified AIC estimator.