Originating as a philosophical quest, the study of personality concerns how individuals differ in thinking, feeling, and behaving. Towards building social machines that work with humans on a daily basis, we are motivated to ask: Do existing LLMs **possess** personalities akin to their human counterparts? If so, how can we **evaluate** them? Further, given this evaluation framework, how can we **induce** a particular personality in a controllable fashion? To answer these three questions, we propose the Machine Personality Inventory (MPI) dataset for evaluating the machine personality; MPI follows standardized personality tests, built upon the Big Five Personality Factors (Big Five) theory and personality assessment inventories. By systematically evaluating LLMs with MPI, we provide the first piece of evidence showing the existence of personality in LLMs. We further devise a Personality Prompting (P^2) method to induce LLMs with a specific personality in a **controllable** manner, capable of producing diverse behaviors. We hope this work sheds light on future studies by adopting personality as the essential guide for various downstream tasks, building human-like and in situ dialogue agents.