Title:RMT Estimator with Adaptive Decision Criteria for Estimating the Number of Signals Based on Random Matrix Theory

Abstract:Estimating the number of signals embedded in noise is a fundamental problem in signal processing. As a classic estimator based on random matrix theory (RMT), the RMT estimator estimates the number of signals via sequentially testing the likelihood of an eigenvalue as arising from a signal or noise for a given over-detection probability. However, it tends to down-estimate the number of signals as weak or even strong signal eigenvalues may be immersed in the bias term among eigenvalues for finite sample size. In order to detect signal eigenvalues immersed in this bias term, we propose an RMT estimator with adaptive decision criterion by incorporating the bias term into the decision criterion of the RMT estimator. Firstly, we analyze the effect of this bias term on the estimation performance of the RMT estimator. Specifically, we derive the analytical formulas for the increased down-estimation probability and the decreased over-estimation probability of the RMT estimator incurred by the bias term among eigenvalues. Then, the RMT-ADC estimator can adaptively determine whether the eigenvalue being tested is arising from a signal or from noise, and can also determine whether the bias term among eigenvalues should be incorporated into the decision criterion of the RMT estimator or not. Therefore, the RMT-ADC estimator can successfully detect signal eigenvalues immersed in this bias term, and thus avoids both the higher down-estimation probability and the higher over-estimation of the RMT estimator. Finally, simulation results are presented to show that the proposedRMT-ADC estimator can successfully detect the signal eigenvalues immersed in the bias term among eigenvalues and significantly outperforms the existing estimators in all cases.