The rapid development of location-based social networks (LBSNs) provides a substantial amount of image data which not only reveals visual contents of POIs but also users’ visual preferences. We argue that the combination of visual content and other side information (e.g., geographical influence) can lead to a more accurate and personalized recommendation performance. In this paper, we enhance POI recommendation by proposing a unified framework named VCG, which incorporates visual contents and geographical influence in LBSNs. Specifically, we employ an overlapping community detection method to capture heterogeneous relations between POIs. Then high-level visual features are leveraged to model two types of POI relations in communities (i.e., POI–POI and POI–Community). Moreover, we design an objective function with social regularization terms based on weighted matrix factorization to learn latent vectors of users, POIs and communities. In terms of geographical influence, an improved power-law probabilistic model is proposed to discover users’ geographical preferences. Finally, we develop a fused POI recommendation framework which joints user preferences with POI visual relations and geographical influences. We evaluate VCG on two real-world datasets: Yelp and Breadtrip, and experimental results show that the proposed framework is effective for the POI recommendation task.