In this section, we describe the results of the experiments done on single item recommendation (Section
5.1), slate top-K recommendation (Section
5.2), and slate reranking (Section
5.3). Again, we remind the reader that the goal of these experiments is to demonstrate the possibilities and challenges of the environments derived from SARDINE, rather than to create a benchmark for state-of-the-art approaches on a limited set of environments. These experiments should be seen as a starting point for researchers and practitioners to further investigate the specific scenarios and approaches of their interest.
5.1 Experiments on Single Item Recommendation
We performed experiments on three
SingleItem environments whose characteristics are recalled below (see Section
4.1 for a more detailed description). In
SingleItem-Static, we consider an easy, static recommendation scenario in a fully observable setting and with low uncertainty to validate that learned agents can reach optimal performance.
SingleItem-BoredInf adds boredom and influence mechanisms to
SingleItem-Static, increasing the difficulty of the environment. For
SingleItem-PartialObs, we start as well from
SingleItem-Static but change it to a POMDP setting. The results on the
SingleItem environments are given in Figure
2 and discussed below.
SingleItem-Static. The validation return over the different training steps on
SingleItem-Static is plotted in Figure
2(a). In this experiment, we compared SAC and REINFORCE agents against the Greedy Oracle and Random baselines. In this specific setting, the Greedy Oracle is by design optimal due to the absence of long-term mechanisms (boredom or influence). It is therefore not surprising that the Greedy Oracle achieves a return of 100, meaning that all recommended items in the session have been clicked. However, it is interesting to note that among the learned agents, SAC fares much better than REINFORCE. Indeed, the former is able to reach optimal performance (or very close to it) after only 200,000 steps, whereas the latter struggles to close the gap. This difference might be explained by the fact that SAC exploits the ideal item embeddings whereas REINFORCE simply selects actions through a softmax over items. Additionally, SAC is generally a better performing RL agent than REINFORCE in most cases, due to a better bias-variance trade-off. Nonetheless, it is reassuring to see that in this simple environment, a learned agent such as SAC is able to easily find the optimal policy.
SingleItem-BoredInf. We now turn to the more challenging
SingleItem-BoredInf environment, which includes boredom and influence mechanisms. The results according to the return and boredom metrics are illustrated in Figures
2(c) and
2(d), respectively. This environment corresponds to typical RL-based recommendation in an MDP setting, and we expect RL agents to be able to beat a myopic approach such as the Greedy Oracle. Indeed, the Greedy Oracle is no longer optimal here due to the introduction of long-term mechanisms. This is confirmed by the results in the plots, which show that the Greedy Oracle only yields a return of around 50, and a boredom of around 50 as well. This means that for 50% of the steps the user is in a bored state, and for the remaining 50% the recommendation leads to a click. Turning to the RL agents, we first see that REINFORCE struggles to learn an effective policy and remains inferior to the Greedy Oracle in terms of return. However, SAC is able to provide high-quality recommendations, with a return close to 70 even after only 50,000 training steps. SAC is also able to recommend diversified items, as its low (and diminishing) boredom confirms. However, the boredom of REINFORCE increases steadily as return increases, showing that its policy is still in an accuracy improvement stage and is not favoring result diversification.
SingleItem-PartialObs. This environment is static like
SingleItem-Static, but we consider here a POMDP setting, i.e., partial state observability. This corresponds to the typical sequential recommendation scenario based on offline feedback, where recommendations have no causal effect on user behavior [
10]. Due to the partial observability, RL agents require a state encoder to convert the observations returned by the environment into a state that can be exploited by the agent. We consider both a GRU and a transformer state encoder, which we test in combination with a SAC agent as it obtained convincing results on
SingleItem-Static. The results are shown in Figure
2(b). Here, the Greedy Oracle is still optimal as it has access to the true user state (and thus is not affected by the POMDP setting). The partial observability does have an impact on the SAC agents, leaving a gap between the 60+ return obtained by these and the 100 return obtained by the Greedy Oracle. This highlights that more research might be needed on state encoders to be able to accurately estimate the true user state in this setting. Comparing the variants of SAC equipped with a GRU and transformer state encoder, we do not notice statistically significant differences between those two in this case.
5.2 Experiments on Slate Top-K Recommendation
We now move on to the experiments performed on slate top-K recommendation, i.e., where the recommendation presented to the user is a list instead of a single item as in
SingleItem environments. We studied four
SlateTopK environments, which are summarized below (and further detailed in Section
4.1).
SlateTopK-Bored and
SlateTopK-BoredInf are both fully observable environments that require agents to do multi-step reasoning to perform well. The difference between the two is that the former only includes a boredom mechanism, whereas the latter additionally integrates an influence mechanism—causing user embeddings to drift based on clicked items and thus making it more difficult to track user interest. We also investigated a partially observable version of
SlateTopK-BoredInf through
SlateTopK-PartialObs, further increasing the difficulty of the task. Finally, we experimented with various levels of uncertainty in the clicking process through the
SlateTopK-Uncertain environments. The results on the
SlateTopK environments are shown in Figures
3,
4,
5, and
6.
SlateTopK-Bored. The results on the
SlateTopK-Bored environment are plotted in Figure
3(a) (for the return) and Figure
3(b) (for the boredom). We compared the Greedy Oracle baseline, which is sub-optimal here, to four learned agents: REINFORCE + Top-K, SAC + Top-K, SAC + GeMS and HAC. Similar to the results observed on
SingleItem environments, we see here that REINFORCE + Top-K fails to reach the performance of the Greedy Oracle, while SAC + Top-K beats by a good margin the Greedy Oracle. HAC and SAC + GeMS are also able to beat the oracle baseline, but by a much smaller margin. In terms of boredom, SAC + Top-K is also the winner with a much lower value. We also report the distribution of item relevance scores for the different methods tested here in Appendix
D. We hypothesize that the superiority of SAC + Top-K, in particular over SAC + GeMS and HAC, is due to the use of ideal item embeddings in this experiment. Indeed, SAC + Top-K directly uses the item embedding space as action space and thus rely heavily on the quality of item representations. To investigate this hypothesis, we repeated the same experiment as shown in Figures
3(a) and
3(b), but we replace the ideal item embeddings used by default in our experiments with item embeddings learned by
matrix factorization (MF).
10 We report the results in Figures
3(c) and
3(d) for the return and boredom metrics, respectively. We observe that changing from ideal to MF embeddings does have a drastic effect on the recommendation performance (measured by the return metric) of SAC + Top-K, which degraded to the level of the Greedy Oracle. The performance of HAC is also greatly impacted—its return dropping even below that of REINFORCE Top-K. SAC + GeMS is the approach that underwent the smallest performance drop in comparison to the ideal embedding case. This suggests that this latter approach might overall be more robust to sub-optimality in item embeddings.
In addition to the results described above on
SlateTopK-Bored, we also investigated a challenging variant of this environment that is closer to a real-life scenario, where certain topics tend to co-occur and the distribution of topics for items and users is skewed. The setting and the experiments done in this environment are further detailed in Appendix
B.
SlateTopK-BoredInf. On the
SlateTopK-BoredInf environment, which adds an influence mechanism to
SlateTopK-Bored with ideal embeddings, we observe similar trends as this latter environment. The return and boredom results are given in Figures
4(a) and
4(b), respectively. One notable difference with
SlateTopK-Bored, however, is that in
SlateTopK-BoredInf HAC fails to beat the Greedy Oracle and gets a return that is significantly worse than that of SAC + GeMS. This might be explained by the fact that HAC integrates a supervised click prediction loss, which may hinder the model performance due to the greater dynamics in the user embedding caused by the influence drift.
SlateTopK-PartialObs. The results on the
SlateTopK-PartialObs environment, which increases
SlateTopK-BoredInf’s challenge with partial observability, are shown in Figure
5(a) (for return) and Figure
6(a) (for boredom). Given the superior performance of SAC + Top-K on
SlateTopK-BoredInf, we focus here on variants of this method based on a GRU or a transformer state encoder. In this setting, we observe that the performance of the transformer variant leads on most training steps to a significant improvement in terms of return over the GRU variant. This result goes in line with previous findings on state encoders for RL-based recommendation [
17]. However, both SAC + Top-K variants fail to beat the Greedy Oracle baseline, highlighting the difficulty of this environment and showing that additional efforts on the agent and/or state encoder might be needed to achieve high-quality recommendation.
SlateTopK-Uncertain. Starting from
SlateTopK-PartialObs, we varied the level of uncertainty in the clicks through the
\(\lambda\) scale hyperparameter in the simulator’s relevance function. In particular, we compared the setting of
SlateTopK-PartialObs with low uncertainty (
\(\lambda = 100\) ) to different
SlateTopK-Uncertain environments with medium uncertainty (
\(\lambda = 10\) ), high uncertainty (
\(\lambda = 5\) ), and very high uncertainty (
\(\lambda = 2\) )—which we will refer to as
SlateTopK-Uncertain10,
SlateTopK-Uncertain5, and
SlateTopK-Uncertain2 for simplicity. The return and boredom results in these environments are illustrated in Figures
5 and
6, respectively. Comparing the return on
SlateTopK-PartialObs (Figure
5(a)) to
SlateTopK-Uncertain10 (Figure
5(b)), we observe that the gap between the SAC + Top-K transformer and GRU variants increases. Indeed, while the overall performance of SAC + Top-K GRU slightly decreases with the uncertainty increase, we see that SAC + Top-K transformer is able to maintain its performance at around 125 at 500,000 steps. This suggests that the transformer state encoder is more robust to a medium level of uncertainty. When we increase the uncertainty to a high level in
SlateTopK-Uncertain5 (Figure
5(c)), we notice that the SAC Top-K variants beat the Greedy Oracle baseline, and that the gap between the Random baseline and the Greedy oracle shrinks. This is explained by the fact that with more stochasticity in the clicking process, less relevant items get more clicks—which reduces the advantage of the greedily optimal recommendations from the Greedy Oracle. Clicks on more varied items also means that user boredom is less likely to be triggered, which is confirmed by the comparison of the boredom scores of the SAC Top-K variants across Figures
6(a),
6(b), and
6(c). When the uncertainty level is further increased in
SlateTopK-Uncertain2, we observe that the environment rewards random recommendations more than the accurate recommendations from the Greedy Oracle, as shown in Figure
5(d). This is, again, explained by the fact that less relevant items lead to a click probability similar to that of relevant items. In this setting, the SAC Top-K variants both perform similarly to the Random baseline and are thus learning to favor more diverse recommendations over accurate ones.
5.3 Experiments on Slate Reranking
With this last set of experiments, we explore the reranking task using two
SlateRerank environments: the static
SlateRerank-Static and the interactive, multi-step
SlateRerank-Bored. We conduct experiments on click modeling according to the following protocol: (i) we generate a dataset of interactions using a certain logging policy, (ii) we train a click model on the generated dataset, and (iii) we rerank the items by decreasing amount of relevance, according to the model. For both environments, we use the reverse-oracle policy, i.e., the policy that orders items by
increasing order of relevance, as the logging policy. It therefore generates a dataset that contains substantial spurious correlations due to position bias. We report the observed return when applying the reranking methods in the live environment in Table
4.
SlateRerank-Static. On the static environment, the Greedy Oracle policy is the optimal policy, while the Reverse Oracle yields minimal return. We can indeed first verify in Table
4 that an online-trained SAC + Top-K, as in Section
5.2, even with full observability of user state and ideal item embeddings, does not beat the greedy oracle.
Second, we can see that a position-based model (PBM) correctly identifies the biases in the logged data and almost reaches the performance of the oracle policy, while the naive document click-through rate model (dCTR) model fails to do so, and barely improves on the Reverse Oracle policy it was trained on. This result does not come as a surprise, since the underlying user click model in the SlateRerank environments is also a position-based model. We can nonetheless verify that the learned propensities, i.e., observation probabilities at each rank, match the true propensities of the simulator. We therefore compute the mean-squared error (MSE) of the normalized propensities, i.e., where the probability of observation at the first position is set to 1, and we find that the learned PBM’s propensities have an MSE of 0.570. That indicates that despite documents being correctly ordered, the learned model does not fully match the underlying model.
The experiments on
SlateRerank-Static call for further experiments with different underlying user click models and candidate click models, e.g., as done in Reference [
9], to investigate the performance of click models under the more realistic settings of model mismatch. While we leave this for readers to experiment with, we turn to another natural extension that is, to the best of our knowledge, unexplored, and that SARDINE enables.
SlateRerank-Bored. In this interactive environment, the Greedy Oracle policy is not optimal anymore, because the agent must trade off the accuracy and diversity of the most-exposed topics. Indeed, we can see in Table
4 that an online-trained SAC + Top-K agent beats the Greedy Oracle. This environment therefore constitutes a testbed for (offline) RL agents with biased feedback, and notably the combination of RL and click modeling.
Another important difference that comes with this interactive environment is the fact that the logged data may appear relatively noisier to a click model, as the click/skip feedback can be explained by something else than relevance and position: the boredom. While the boredom information is contained in the ideal user state we use for click model training and the model should therefore in theory be able to correctly identify biases, we expect the training process to be harder. We observe what seems to be a slight degradation of relative performance, compared to the static environment. Indeed, while the PBM managed to fill 98% of the gap between the logging policy and the oracle policy on the Static environment, it only fills 89% of the gap on the Bored environment. But the extent of the degradation is most apparent when we compare the propensities learned by the model in this new dynamic environment. The MSE now increases to 0.915, compared to 0.570 in the static case. This suggests that using the learned propensities of the model in downstream tasks, e.g., counterfactual learning-to-rank, fairness or reinforcement learning, is likely to lead to imperfect and biased policies.
Effectively using the user behavior learned by click models in a dynamic and interactive environment with, e.g., reinforcement learning, including when the learned variables are imperfect, is to the best of our knowledge still an open question. Our proposed simulator offers the possibility to study this topic in an interpretable and controllable way.