Revisiting Negative Sampling vs. Non-sampling in Implicit Recommendation

Early recommendation methods [36, 37] were mainly designed to model users’ explicit feedback such as ratings to movies. However, implicit feedback is actually easier to collect in real-world scenarios, such as clicks in news portals, purchases in e-commerce, and views on online video applications. For recommendation with implicit feedback data, a lot of methods were proposed before [10, 30, 33, 52]. Since truly negative is mixed with unlabeled positive, these methods differ from each other in how to use unobserved data. Specifically, Hu et al. [33] proposed a non-sampling-based method, WMF, which treats all unobserved items as negative samples and assigns them a lower constant weight. Then several efforts [32, 42] improve WMF by applying different weighting strategies based on whether the unobserved items are indeed negative ones. Different from non-sampling methods, Rendle et al. [52] proposed a sampling-based method, BPR, which optimizes the MF model based on the relative preference of a user to positive and negative items. The pairwise learning strategy with negative sampling has been widely adopted to optimize recommendation models [6, 12, 30, 59] and has become a dominant technique in recommendation.

With the development of deep learning techniques, there is a lot of work exploiting different neural networks for recommender systems. The work [30] presented a Neural Collaborative Filtering (NCF) framework to jointly learn a matrix factorization and a feedforward neural network for implicit Top-K recommendation. NCF has been widely extended for different recommendation scenarios [12, 26, 64]. In recent years, it has become a trend to explore the application of advanced deep learning architectures for recommendation tasks, such as leveraging attention mechanisms [5, 12, 71], Recurrent Neural Network (RNN) [46], Convolutional Neural Network (CNN) [28, 81], GANs [29, 61], and Graph Neural Networks (GNNs) [20, 65]. Specifically, [65] proposed NGCF to model high-order connections by propagating embeddings on the user-item interaction graph. LightGCN [27] is an extended version of NGCF by removing the feature transformation and non-linear activation function to improve the performance of the recommendation task. Besides the above methods that mainly use negative sampling for model learning, there are also some neural recommendation models based on non-sampling learning. For example, Chen et al. derived a flexible non-sampling loss and designed several efficient non-sampling neural models for various recommendation scenarios [7, 9, 10, 11].

There are two learning strategies that have been proposed before to learn from implicit feedback: negative sampling strategy [6, 30, 52] and non-sampling strategy [16, 33, 42, 67].

The negative sampling strategy samples negative instances from the unobserved data. Through sampling, the scale of training samples is greatly reduced; therefore, the training process is more efficient [30]. Negative sampling has been widely applied in many recommendation models, including traditional recommendation models like BPR [52] and neural models like NCF [30]. The most popular and widely used sampling strategy is uniform sampling (also called random sampling). However, uniform sampling usually samples uninformative training instances, which usually make a limited contribution to update the model. To deal with this problem, many methods have been proposed in recent literature to replace the uniform sampling with other better samplers [17, 41, 51, 61] to improve the quality of negative samples. For example, [41, 76] proposed to sample negative examples based on the popularity of items. [17, 51] proposed to sample hard negative instances with higher prediction scores. [74] proposed a sampling-bias-corrected algorithm for estimating item frequency from streaming data. [72] proposed to use both batch and uniformly sampled negatives to tackle the selection bias of implicit recommendation. There are also some GAN-based methods in which the sampling probability will adaptively evolve by optimizing adversarial objectives [29, 35, 49, 61]. Another kind of method samples the negative instances based on the structure of the graph. For example, [66] incorporated a knowledge graph into the negative sampling process to sample high-quality negative items. [75] sampled the negative node according to their pagerank scores. [34] proposed a MixGCF method with the hop mixing and positive mixing strategies for GNN-based recommendation models. However, the above methods will suffer from inefficiency issues since the sampling process will dynamically change and usually needs to calculate all instances.

The non-sampling strategy sees all the unobserved data as negative, while assigning them with a lower weight than positive examples. For example, WMF [33] assigned all unobserved entries with a uniform weight. In EALS [32] and ExpoMF [42], the weight of unobserved entries is dependent on item popularity, which is based on the assumption that popular items are easier to be seen by users, so they should be assigned higher weights as negative. A non-sampling strategy has been shown that can leverage the whole data with a potentially better coverage [9, 32, 42, 70], but inefficiency can be an issue. To this end, a number of methods have been developed to accelerate the learning process, including batch-based ALS [1, 32, 33] and mini-batch SGD methods [9, 70]. However, this kind of method is only suitable for the recommendation models with a linear prediction layer [1, 10, 70] and regression loss function. Non-sampling methods have been widely applied in many traditional recommendation models [32, 33, 42] but have few applications in neural recommendation models. Recently, Chen et al. [10] proposed ENMF, which is a representative neural non-sampling recommendation model. To the best of our knowledge, although there are some works on improving negative sampling and non-sampling respectively, the in-depth comparison between them is left insufficiently explored. This is the main concern of this work.

The above methods are all based on discriminative modeling, which explicitly aims to distinguish positive user-item interactions from the negative counterparts. Recently, there has been another line of research that utilizes self-supervised learning for implicit recommendation training [39, 82]. Given a positive user-item interaction \((u, i)\) , the idea is to make representations for u and i similar to each other to encode the preference information. To address the potential model collapse problem, this kind of method generally applies two distinct encoder networks (i.e., online and target networks). Compared to discriminative modeling methods, self-supervised learning methods are easier to collapse into a trivial constant solution, so that the hyper-parameters need to be carefully tuned [24, 39]. To make our work more focused, the methods discussed in this article are all based on discriminative modeling. We leave the exploration of self-supervised learning as future work.

Negative sampling has also been widely used in other domains and tasks of machine learning, such as graph embedding [73], word embedding [45], and network embedding [79]. For example, Word2Vec [45] samples negative samples according to its word frequency, which is similar to the sampling process in recommendation. Later works on network and graph embedding [25, 50, 58] follow this setting. It has also been observed that negative instances with large scores (hard) are more useful for model training [79]. Another recent work on negative sampling of graph learning shows that the negative sampling distribution should be positively but sub-linearly correlated to their positive sampling distribution [73]. There are also some GAN-based methods for the above tasks [2, 3, 22]. For non-sampling-based methods, [8, 40] proposed efficient non-sampling methods for learning knowledge graph embeddings.

Since the implicit recommendation is a different problem, the reliability of sampled negative instances is much harder to guarantee. In this article, we mainly focus on the comparison of negative sampling and non-sampling for the recommendation task and leave the exploration of other tasks as future work.

Table 1 shows the key notations used in this article. We denote user set U (including M users) and item set I (including N items). The implicit feedback data is denoted as an \(M \times N\) binary matrix Y, where \(y(u,i) \in \lbrace 0,1\rbrace\) denotes whether user u has interacted with item i or not. We use \(\mathcal {Y}\) to denote the set of observed entries in Y. Moreover, \({\bf I}_u\) is used to denote the positive item set of user u. Given a target user u, the task of implicit recommendation is to learn user u’s preference based on the implicit feedback data and recommend items that might be of interest to u.

An implicit recommendation model with negative sampling learning strategy generally has three important components: scoring function \(\hat{y}\) , objective function \(\mathcal {L}\) , and negative sampling strategy \(p_{ns}\) [17]. The scoring function \(\hat{y}({\bf p}_u, {\bf q}_i, \Theta)\) predicts the preference of user \(u \in {\bf U}\) to item \(i \in {\bf I}\) based on u’s latent factor \({\bf p}_u\) and i’s latent factor \({\bf q}_i\) . It has been widely studied in previous work, including various models based on Matrix Factorization (MF) [29, 52, 61] and neural networks [26, 27, 30, 63, 65].

For objective function, Bayesian Personalized Ranking (BPR) [52] is widely used in the negative-sampling-based recommendation model, and is also the most representative learning method. The formulation of BPR is as follows:where \(\sigma (x)= \frac{1}{1+e^{-x}}\) is an activation function (sigmoid) and \({\bf Y}\) is the training data. The negative instance \((u, j)\) is sampled through a specific distribution \(p_{ns}(j | u)\) . \(\sigma (\hat{y}(u, i)-\hat{y}(u, j))\) is modeled as the likelihood that a user u prefers item i to item j, which approaches 1 when \(\hat{y}(u,i) \gg \hat{y}(u,j)\) . Minimizing \(\mathcal {L}\) is equal to make the scores of positive user-item interactions larger than negative user-item interactions. \(\ln \sigma (x)\) is actually a differential surrogate function of the Heaviside function, so BPR approximately optimizes the ranking statistic AUC [41].

As the number of training pairs \(|{\bf Y}|\) is usually very large, learning algorithms typically are based on SGD. The gradient of BPR to the model parameters isThe gradient depends on how the scoring model would discriminate between the positive item i and the negative item j for user u. \(\frac{\partial \mathcal {L}(u,i,j)}{\partial \Theta }\) is a probability, which is close to 0 if \(\hat{y}(u, i)\) is to correctly get a larger score than \(\hat{y}(u, j)\) .

From the above gradient, we analyze the influence of negative sampling from both the sampled right examples (i.e., examples that are really negative to user u) and false examples (i.e., examples that are positive to user u in the test set). For a sampled right negative example j, positive item i can be easily distinguished from item j (i.e., \(\sigma (\hat{y}(u, i)-\hat{y}(u, j)) \rightarrow 1\) ) when the model is well trained, so that the pair \((u, i, j)\) will contribute little to the model learning because its gradient vanishes ( \(\frac{\partial \mathcal {L}(u,i,j)}{\partial \Theta } \rightarrow 0\) ). This kind of issue is particularly prominent when the sampling distribution is uniform. In recommender systems, item popularity is typically non-uniform distributed, and overall positive observations have a tailed distribution. It is very likely that the model score \(\hat{y}(u,j)\) of a uniformly sampled item j is smaller than \(\hat{y}(u,i)\) and thus the gradient magnitude is also small. Therefore, the widely used uniform sampling strategy is generally very slow to converge and hard to achieve the optimal performance.

To avoid this, the prediction scores of items are used for sampling in previous work, e.g., AOBPR and BPR-DNS [49, 51]. These kinds of methods are designed to sample more hard instances, i.e., unobserved items with higher prediction scores. However, since items with higher prediction scores are more likely to be actually positive in the test set, these kinds of methods suffer from false examples more markedly, which will hurt the model’s performance and robustness since they are sampled as negative instances during training.

From the above analyses of negative instances in the training of implicit recommendation, we can conclude that the difficulty of negative sampling is how to sample negative examples with large scores to increase the convergence speed and avoid false-negative instances to maintain robustness.

A recommendation model with the non-sampling learning strategy typically creates the training data from \(\mathcal {Y}\) by giving pairs \(y(u, i)\in \mathcal {Y}\) a positive label and all other unobserved interactions \({\bf Y}\backslash \mathcal {Y}\) a negative label:Then the recommendation model is fitted to this data. The widely used non-sampling-based objective function is a weighted regression function, which assigns a training weight to each interaction in the implicit matrix:where \(c(u,i)\) denotes the weight of entry \(y(u,i)\) .

Through the above objective function, the model is optimized to predict the value 1 for elements in \(\mathcal {Y}\) and 0 for the rest. The purpose of this kind of method is to impose the gravity regularizer to penalize the prediction for missing items [41]. One of the weaknesses of this approach is that all candidate ranking items are presented to the recommendation model as negative instances during training, which means that a model with enough expressiveness cannot generate a reasonable ranking list at all as it predicts only 0.

Besides, considering that the number of missing data can be much larger than user interacted items in real applications, it is more desirable to assign the missing data a lower weight to address the class imbalance issue. A weighting strategy for unobserved examples in non-sampling methods plays a similar role to the negative sampling strategy in the sampling-based strategy [32, 42]. The difference would be that with a weighting strategy for unobserved samples all samples are used, whereas with a sampling strategy some samples might never be used during training, which may have an impact in terms of generalizability or may leave the model not trained for certain types of input, lowering its robustness. The methods with non-uniform weighting can model negative instances with a faster coverage but generally require more computational cost due to the dense weight matrix for all user-item pairs. Although some methods have been proposed to accelerate the learning process from the whole data, one challenge remains in existing non-sampling approaches: the fast non-sampling learning methods only suitable for the recommendation models with a linear prediction layer.

Uniform sampling with BPR pair-wise loss function [52] is a most widely used and classical solution for item recommendation with implicit feedback. Uniform negative sampling over unobserved items is also called random sampling, which is based on the simplest uniform proposal. We use \(Q(j|u,i)\) to represent the probability that a negative item j is sampled for a positive pair \((u, i)\) , which is defined aswhere \(N_u=|{\bf I}_u|\) denotes the number of user u’s interacted items.

Although uniform sampling has been successfully applied in numerous recommender applications and for a variety of models, it is shown that convergence slows down considerably if the number of items is large and the overall item popularity is tailed. Both properties are common to most real-world datasets.

AOBPR (Adaptive Oversampling BPR) [51] is designed to improve the uniform sampling strategy by adaptively sampling hard instances, i.e., unobserved items with higher prediction scores, which is hard to discriminate for the algorithm. Intuitively, when a negative item j for a positive pair \((u, i)\) should be sampled, the closer j is to the top, the more informative is j. The sampling distribution of AOBPR is defined aswhere \(\hat{r}(u,j)\) is the rank of item j among all items I using the score function \(\hat{y}(u,j)\) for ordering items, and \(\lambda\) is a hyper-parameter to control the skewness of the distribution.

The limitations of AOBPR are (1) it needs to compute the scores of all items in order to determine their sampling probability, which reduces the advantage of sampling and is time-consuming in the training process, and (2) the items with higher prediction scores are more likely to be positive in the test set, and thus AOBPR is more vulnerable to false samples than uniform sampling.

The motivation of BPR-DNS (Dynamic Negative Sampling) [78] is similar to AOBPR, which also tries to adaptively sample hard instances. Different from AOBPR, DNS first randomly selects a set of negative samples, then uses the item with the highest prediction score for model optimization. The training process of DNS is shown in Algorithm 1.

The limitation of BPR-DNS is that it is also vulnerable to false samples since items with higher prediction scores are more likely to be positive in the test set. Compared to AOBPR, the advantage of BPR-DNS is that it only needs to compute the scores of n items for each sampling, and a small number of n (generally no more than 32) is enough [78].

IRGAN [61], a GAN-style negative sampling method, applies a mini-max game for information retrieval tasks like implicit recommendation. Specifically, IRGAN includes two components: a generator G and a discriminator D. The generator is to produce more and more difficult negative samples, and the discriminator is to minimize the discrimination objective function \(\mathcal {J}(D, G)\) . For recommendation with implicit feedback data, IRGAN optimizes the following objective function:where \(P_{\text{pos }}(\cdot \mid u)\) is a positive relevance distribution and \(P_{G}(\cdot \mid u)\) is a probability distribution used to generate negative instances. \(D(i\mid u)\) estimates the probability of user u to item i. When the discriminator D and the generator G are well trained, G is used for the prediction of recommendation.

Although the GAN-style methods have shown promising results for discovering informative negative samples, it is usually very time-consuming to generate negative instances from the generator G, which limits its application ability for large-scale datasets.

SRNS (Simplified and Robust Negative Sampling) [17] is a recently proposed negative sampling method, which is based on the finding that false-negative samples (underlying positive) generally have large prediction scores for many training epochs. SRNS captures the dynamic sampling distribution of negative samples through a memory-based component. To evaluate the quality of negative samples, SRNS also proposes a high-variance-based strategy. The training process of SRNS is shown in Algorithm 2.

The variance-based sampling strategy is proposed to avoid false-negative instances by preferring high-variance candidates, which is defined aswhere \(P_{\mathrm{pos}}(k \mid u, i)=\mathrm{sigmoid} (\hat{y}(u,i) -\hat{y}(u,k))\) , \(\operatorname{std}[P_{\mathrm{pos}}(k \mid u, i)]\) denotes the prediction variance in the latest few epochs. \(\alpha _{t}\) is a hyper-parameter to control the importance of the variance.

The memory strategy is to dynamically update \(\mathcal {M}_{u}\) while including more hard negative samples. The new \(\mathcal {M}_{u}\) is updated by sampling \(S_1\) instances from an extended memory that merges the old \(\mathcal {M}_{u}\) and a set of randomly sampled instances. The sampling probability distribution is as follows:where \(\tau\) is a temperature parameter and a lower \(\tau\) would make \(Q(j|u,\mathcal {M}_{u})\) pay more attention to large-scored instances.

The main training cost of SRNS comes from the above-introduced variance-based sampling and score-based memory update. Specifically, it requires to compute \(S_1\ +\ S_2\) candidates for each positive instance and then sample \(S_1\) of them based on computed scores [17]. According to the original paper, small numbers of \(S_1\) and \(S_2\) are generally sufficient, which makes SRNS more efficient than methods requiring computing all items’ scores.

Different from sampling-based learning methods that mainly focus on improving the quality of negative samples, the key of non-sampling learning is to assign proper weights for unobserved user-item interactions. WMF [33] is a well-known non-sampling learning method, which assigns all unobserved user-item interactions with the same uniform weight:where \(c_0\) and \(c_1\) are hyper-parameters that need to be tuned according to different datasets. To simplify the tuning process, \(c_1\) is usually set to a value of 1, while \(c_0\) is set as a smaller number than \(c_1\) to address the imbalanced optimization issue [33]. For instance, in our experiments, \(c_0\) is selected from [0.001, 0.005, 0.01, 0.05, 0.1, 0.3, 0.7] and is set to 0.5, 0.05, and 0.05 for movielens-1m, pinterest, and yelp2018 datasets. Generally, this parameter is related to the sparsity of data. If the data is more sparse, then a smaller value of \(c_0\) may achieve a better performance. Previous studies [18, 33, 62] have demonstrated that this strategy shows better performance than using the same weight for all user-item interactions in the recommendation task.

The uniform weight strategy assumes that all unobserved user-item interactions have the same level of negative signal, which is too simple to model real-world scenarios. It is easy to understand that a popular item has more chance to be seen by users, and thus it should have a higher weight as negative if not interacted by users [32, 42]. Specifically, the EALS [32] method assigns unobserred user-item interactions with non-uniform weights according to item popularity:where \(c_{v}^-\) is defined aswhere \(\mathcal {Y}_i\) denotes the positive interactions of i, \(m_v\) denotes the popularity (frequency) of item v in \({\bf Y}\) , \(c_{0}\) determines the overall weight of unobserved data, and x controls the significance level of popular items over unpopular ones. To simplify the tuning process, x is usually set to a value of 0.5 as suggested in [32].

The difficulty of applying non-sampling learning for implicit recommendation lies in the expensive computational cost. For example, the complexity of computing Equation (4) is \(O(|{\bf U}||{\bf I}|d)\) , which is generally unaffordable since in the real world \(|{\bf U}||{\bf I}|\) can easily reach to billion level. Several methods have been proposed [9, 32, 70, 77] to address the inefficiency issue of non-sampling learning. Specifically, recent studies [9, 10] propose an efficient loss for two-tower-based recommendation models, and prove that for a generalized matrix factorization framework whose prediction function is Equation (13), the gradient of loss Equation (4) is exactly equal to that of Equation (14) if the weight \(c(u,i)\) is simplified to \(c_i\) :where \(\mathbf {p}_u \in \mathbb {R}^d\) and \(\mathbf {q}_i\in \mathbb {R}^d\) are embeddings of user u and item i, \(\odot\) denotes the element-wise product, and \(\mathbf {h} \in \mathbb {R}^d\) is the prediction vector.

The complexity of Equation (14) is \(O((|{\bf U}|+|{\bf I}|)d^2+ |\mathcal {Y}|d),\) while that of Equation (4) is \(O(|{\bf U}||{\bf I}|d)\) . Since \(|\mathcal {Y}|\) is the number of observed user-item interactions and \(|\mathcal {Y}|\ll |{\bf U}||{\bf I}|\) in practice, the complexity is greatly reduced. The proof of this method can be found in [9, 10]. To avoid repetition, it is omitted here.

Note that this method does not directly calculate the scores of all items. Instead, it reformulates the loss over all negative instances through a partition and a decouple operation to achieve speedup. As such, it cannot be applied to the above sampling-based methods, which require computing a high number of item scores.

The above efficient loss can also be applied to a common matrix factorization recommendation model (i.e., \(\hat{y}{(u,i)}=\mathbf {p}_{u}^T \mathbf {q}_{i}\) ). It is used in our experiment to train non-sampling methods WMF and EALS.

We first make a performance comparison between the above-introduced negative sampling and non-sampling learning methods. The method ItemKNN [56] is also added as a basic benchmark. For fair comparison, all methods are integrated into a common matrix factorization recommendation model (i.e., \(\hat{y}{(u,i)}=\mathbf {p}_{u}^T \mathbf {q}_{i}\) ). We re-implement BPR-Uniform, BPR-DNS, SRNS, WMF, and EALS with TensorFlow. For ItemKNN and AOBPR, we use the implementation in LibRec.⁵ For IRGAN, we use the authors’ released code.⁶ For sampling-based methods, we sample one negative instance for each positive instance, which is a widely used setting of previous work [27, 30, 52, 61]. The impact of sampling number is explored in Section 5.4. To evaluate on different recommendation lengths, we investigate the top-K performance with K setting to [5, 10, 20] in our experiments. The results of the comparison of different methods are shown in Table 4. From the table, we have the following key findings:

Many previous studies only focused on obtaining better results but ignored the computational efficiency [57]. In real-word systems, training efficiency is also an important factor that needs to be considered. In this section, we conduct an experiment to show the training efficiencies of representative negative sampling and non-sampling methods. All the compared methods are re-implemented with Tensorflow and are run on the same machine for a fair comparison. The training time results are shown in Table 5. Note that these comparison models all have the same network structure but use different learning strategies. We have the following key observations:

BPR pair-wise loss (Equation (1)) is the most widely used sampling-based learning strategy. However, existing BPR learning-based methods generally sample one negative instance for each positive user-item pair [6, 27, 52, 65]. The impact of negative sampling number has been largely ignored by existing studies. Here we investigate how the performance changes as the number of negative samples increases. The BPR pair-wise loss with multiple negative samples is calculated as follows:where \(\sigma (x)= \frac{1}{1+e^{-x}}\) is a sigmoid function, \({\bf Y}\) is the training data, and \({\bf I}_u\) denotes the positive item set of user u. It repeats the positive instance \((u,i)\) multiple times to make it get a higher score than every negative item.

Figure 2 shows the performance of BPR-Uniform when varying the number of negative samples on the four datasets. For comparison, we also report the performance of each training epoch. We show the results on Recall@20 metrics in this section. From the figure, we can obviously find that the number of negative samples does matter for training the recommendation model with BPR loss. Generally, it is beneficial to use more negative samples. For example, in Figure 2, sampling multiple negative instances for each positive pair outperforms using only one negative instance, and can even achieve similar results as non-sampling methods on the Movielens-1m dataset. Moreover, the results show that sampling more negative instances leads to faster convergence. Note that in this section we take BPR-Uniform as an example to show the impact of negative sampling number. For other negative sampling methods, similar performance can also be observed [34]. This is because when sampling more negative items, hard negative samples are more probable to be included and provide more valuable gradient updates. The results suggest that sampling more negative instances for each positive pair appears to be a promising setting for the recommendation model with BPR loss to gain higher performance. This also shows the potential of using non-sampling learning for improving the performance of recommendation systems. We therefore call for more future considerations toward the above settings when evaluating a newly proposed recommendation model.

To answer research question 3, we further compare the state-of-the-art recommendation models NGCF [65] and LightGCN [27] with both sampling and non-sampling learning (with efficient loss) strategies to explore how negative sampling and non-sampling learning boost recommendation performance. The compared models are introduced as follows:

We also compare with more advanced sampling-based approaches, including the GAN-based method AdvIR and graph-based methods MCNS and MixGCF, as follows:

To reduce the experiment workload and keep the comparison fair, we closely follow the settings of the MixGCF work [34]. The datasets Yelp2018 and Alibaba are exactly the same as the MixGCF work used, so we directly use the results of AdvIR, MCNS, and MixGCF in the MixGCF paper.

Table 6 shows the performance of the compared methods. From this table, we can make the following observations:

Evaluating recommender systems properly has been realized to be difficult since it relies heavily on empirical results [54]. Recently, several critical studies have found that the improvement achieved with some complex models was only observed because the chosen baselines were weak or the parameters were not properly optimized [14, 15, 44, 54]. In this article, we conduct thorough comparisons between negative sampling and non-sampling learning with careful setup of various representative methods. Although our empirical findings may not generalize to other tasks, we reveal the facts that a simple recommendation model with non-sampling learning can outperform many advanced sampling-based methods, which are usually neglected in previous studies since recently it has become common in research papers that only sampling-based baselines are used to compare with newly proposed techniques [27, 65, 68, 69]. These results also encourage our community to revisit the previously proposed recommendation models with fine-tuned training settings to better investigate their potential performance.

Due to the difficulty of evaluating recommender systems, different works usually report inconsistent results [15, 53], which makes the performance of existing methods not well understood. However, it is difficult to achieve reliable experiments by authors of a single paper, which requires a community effort. As such, we believe that a suite of benchmark datasets and well-tuned baselines should be further developed by our community.

From previous studies, we can see that existing neural recommendation models generally rely on uniformly negative sampling to support efficient training. However, our findings have shown that uniformly negative sampling will lead to suboptimal performance. In fact, the recommendation model has been well studied by a large number of recent works, while the learning strategy usually becomes the bottleneck that limits the recommendation performance. Although some studies have proposed to replace uniform sampling with other sampling methods [17, 41, 51, 61] to improve the quality of negative samples, they usually only integrate the proposed samplers into simple recommendation models like matrix factorization. The main reason is that the above methods cannot meet the requirements of efficiency and effectiveness at the same time (see Table 5). For example, several state-of-the-art methods use complex structures such as GAN [61] to generate negative instances, which has posed a new challenge on model efficiency. In this case, these methods can be hardly incorporated with state-of-the-art recommendation models such as GCN models. How can we efficiently sample informative negative instances remains an important research question and deserves further exploration.

Recently, some recommendation studies also tried to apply the non-sampling learning strategy for model optimization. The results reported in those studies are consistent with our experiments, showing the superior ability of non-sampling learning for recommendation tasks. However, existing efficient non-sampling learning methods are only suitable for the recommendation models with a linear prediction layer, as shown in Figure 3, which limits the scalability and flexibility of model design. It would be beneficial if this kind of method can be applied to non-linear structures. Although challenging, it is a promising future work to propose a general efficient non-sampling learning method.

Some recent studies have pointed out the positive effects of linear prediction [53]: (1) it is relevant to the industry as it is more applicable, (2) it simplifies the modeling and learning process, and (3) it has better alignment with other research tasks such as image models and natural language processing where the dot product is usually used. Some state-of-the-art recommendation models also adopt linear prediction for combining embeddings, such as NGCF [65], LightGCN [27], and KGAT [63]. Therefore, except for extending non-sampliong to non-linear prediction structures, other promising directions to improve non-sampling recommendation models include (1) designing better embedding layers of users and items (f and g in Figure 2) through graph learning or causal modeling; (2) leveraging content features such as context information, review, social connections, and knowledge graph, and (3) optimizing the model with multiple objective functions and multi-task learning.

Multi-task learning is to perform joint training on different but correlated tasks, in order to obtain a better model for each task [60]. Recently, multi-task learning has been widely applied for learning a recommender system with side information such as social connections, knowledge graphs, and users’ multiple behaviors [8, 11, 21]. However, although negative sampling has been widely applied in previous recommendation work, it is still reasonable to argue that existing sampling methods are not very suitable for optimizing a multi-task model. Specifically, to generate a training batch, sampling methods need to sample negative instances for each task. This will produce much larger randomness than single-task learning and would inevitably lead to information loss. The poor performance of negative sampling for multi-task learning has been observed in previous work [7, 11]. Therefore, it will be interesting and valuable to design better and suitable sampling methods for multi-task learning.

Most real-world recommendation systems [80] contain two stages: candidate generation and ranking. For very large systems that contain billions of items and users, both existing negative sampling and non-sampling methods are hard to directly apply due to the huge time or space complexity. In this case, a pre-processing process of candidate generation is usually applied first. For example, [43] uses co-occurrences of items to generate candidates, [19] applies a random walk on a (co-occurrence) graph, and [13] describes a hybrid approach using a mixture of features.

In practical recommender systems where new users, items, and interactions are continuously streaming in, it is important to update the model in real time to best serve users. For non-sampling methods, a commonly used online learning strategy is incremental learning [32]. Given a new user-item interaction ( \(u,i\) ), incremental learning only performs optimization steps for \({\bf p}_u\) and \({\bf q}_i\) . The assumption is that new interactions should not change the overall parameters too much, but should change the embeddings of u and i significantly. For negative sampling, the advantage is that sampling-based models are easy to get parameters updated continuously with new data. However, the problem is that existing state-of-the-art sampling methods cannot meet the requirements on effectiveness and efficiency at the same time. This, as we have discussed, deserves further exploration.