Voice, as a convenient and efficient way of information delivery, has a significant advantage over the conventional keyboard-based input methods, especially on small mobile devices such as smartphones and smartwatches. However, the human voice could often be exposed to the public, which allows an attacker to quickly collect sound samples of targeted victims and further launch voice impersonation attacks to spoof those voice-based applications. In this paper, we propose the design and implementation of a robust software-only voice impersonation defense system, which is tailored for mobile platforms and can be easily integrated with existing off-the-shelf smart devices. In our system, we explore magnetic field emitted from loudspeakers as the essential characteristic for detecting machine-based voice impersonation attacks. Furthermore, we use a state-of-the-art automatic speaker verification system to defend against human imitation attacks. Finally, our evaluation results show that our system achieves simultaneously high accuracy (100%) and low equal error rates (EERs) (0%) in detecting the machine-based voice impersonation attack on smartphones.