Since the beginning of 2020, the coronavirus disease 2019 (COVID-19) has spread rapidly in the city of Wuhan, P.R. China, and subsequently, across the world. The swift spread of the virus is largely attributed to its stealth transmissions in which infected patients may be asymptomatic. Undetected transmissions present a remarkable challenge for the containment of the virus and pose an appalling threat to the public. An urgent question that has been asked by the public is "Should I be tested for COVID-19 if I am sick?". While different regions established their own criteria for screening infected cases, the screening criteria have been modified based on new evidence and understanding of the virus as well as the availability of resources. The shortage of test kits and medical personnel has considerably limited our ability to do as many tests as possible. Public health officials and clinicians are facing a dilemma of balancing the limited resources and unlimited demands. On one hand, they are striving to achieve the best outcome by optimizing the usage of the scant resources. On the other hand, they are challenged by the patients' frustrations and anxieties, stemming from the concerns of not being tested for COVID-19 for not meeting the definition of PUI (person under investigation). In this paper, we evaluate the situation from the statistical viewpoint by factoring into the considerations of the uncertainty and inaccuracy of the test, an issue that is often overlooked by the general public. We aim to shed light on the tough situation by providing evidence-based reasoning from the statistical angle, and we expect this examination will help the general public understand and assess the situation rationally. Most importantly, the development offers recommendations for physicians to make sensible evaluations to optimally use the limited resources for the best medical outcome.