Recently, generative adversarial networks (GANs) can be used to generate the photo-realistic image from a low-dimension random noise. It is very dangerous that the synthesized or generated image is used on inappropriate contents in social media network. In order to successfully detect such fake image, an effective and efficient image forgery detector is desired. However, conventional image forgery detectors are failed to recognize the synthesized or generated images by using GAN-based generator since they are all generated but manipulation from the source. Therefore, we propose a deep learning-based approach to detect the fake image by combining the contrastive loss. First, several state-of-the-art GANs will be collected to generate the fake-real image pairs. Then, the contrastive will be used on the proposed common fake feature network (CFFN)to learn the discriminative feature between the fake image and real image (i.e., paired information). Finally, a smaller network will be concatenated to the CFFN to determine whether the feature of the input image is fake or real. Experimental results demonstrated that the proposed method significantly outperforms other state-of-the-art fake image detectors.