Pointwise prediction and Learning to Rank (L2R) are both widely used in recommender systems. Currently, these two types of approaches are often considered independently, and most existing efforts utilize them separately. Unfortunately, pointwise prediction tends to overfit the training data while L2R is more prone to higher variance, and both of them suffer one-class problems using implicit feedback. Therefore, we propose a new framework called CPL, where pointwise prediction and L2R are inherently combined to discriminate user preferences on unobserved items, to improve the performance of top-N recommendations. To verify the effectiveness of CPL, an instantiation of CPL, which is named CPLmg, is introduced. CPLmg is based on two components, i.e., FSLIM (Factorized Sparse LInear Method) and GAPfm (Graded Average Precision factor model), to perform pointwise prediction and L2R, respectively. The low-rank users’ and item’s latent factor matrices act as a bridge between FSLIM and GAPfm. Moreover, FSLIM dynamically rates an unobserved item for a user based on its similarity with observed items. These pseudo ratings are further utilized with a confidence score to rank items in GAPfm. Extensive experiments on two datasets show that CPLmg significantly outperforms the baselines.