Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning

Several approaches in HitL RL allow agents to leverage human feedback to adapt or learn new behavior. Learning from demonstration (LfD) Argall et al. (2009) is one such methodology that allows a human to provide examples of desired agent behavior. Human demonstration data has been used to shape the environment’s reward function Brys et al. (2015), develop a reward function from scratch Abbeel and Ng (2004), and bias the agent’s policy towards certain actions Taylor et al. (2011). Although demonstrations can be a rich source of feedback, they are often expensive to obtain and may require domain experts Dragan and Srinivasa (2012).

Another approach is learning from preference-based feedback where a teacher provides preferences between two or more sets of agent behavior Christiano et al. (2017). Preference learning has been popularized in recent years as it can require less effort and expertise compared to providing demonstrations. To further reduce the amount of human interaction required, several strategies have been introduced. Recent work has combined preferences with demonstrations Ibarz et al. (2018); Bıyık et al. (2022), used unsupervised pre-training for policy initialization Lee et al. (2021), integrated semi-supervised learning and data augmentation techniques Park et al. (2022), applied uncertainty-based exploration strategies Liang et al. (2022), and leveraged labeled data from multiple tasks via a meta-learning approach Hejna III and Sadigh (2023). Despite its popularity, some argue that comparison feedback might not capture the full intricacies of human preferences, as oftentimes the human is limited to choosing between two options Daniel et al. (2014); White et al. (2023).

As a result, another body of work focuses on learning from scalar feedback where human teachers can provide scalar signals to evaluate an agent’s behavior Knox and Stone (2009); Griffith et al. (2013); Loftin et al. (2016); MacGlashan et al. (2017); Warnell et al. (2018); White et al. (2023). Several works use scalar feedback to learn a reward model via regression Daniel et al. (2014); Cui and Niekum (2018); Cabi et al. (2020). Similarly, other works learn a human reinforcement function via regression Knox and Stone (2009, 2013); Warnell et al. (2018). In this setting, a human is assumed to provide scalar feedback that is representative of a behavior’s long-term value. A human reinforcement function is distinct from a reward function, and more closely resembles an action-value function in RL.

Despite the growing advances in HitL RL, there is no work, to the best of our knowledge, in understanding how reward-free, sub-optimal data can be leveraged to improve the feedback efficiency of preference- and scalar-based approaches.

SDP aims to leverage sub-optimal data for scalar- and preference-based HitL RL algorithms. However, learning from low-quality data or negative examples has been applied in other areas of reinforcement learning and imitation learning Chen et al. (2021); Tangkaratt et al. (2021). In standard RL, several works use sub-optimal demonstrations to initialize a policy Taylor et al. (2011); Hester et al. (2018); Gao et al. (2019). In offline RL, some approaches leverage sub-optimal transitions from multiple tasks and either assign reward labels according to the reward function of the target task Singh et al. (2020) or simply label them the minimum possible environmental reward Yu et al. (2022). In goal-conditioned RL, Hindsight-Experience-Replay (HER) Andrychowicz et al. (2017) uses failed episodes by treating them as a success with respect to a different goal. In inverse reinforcement learning (IRL) Arora and Doshi (2021), a constrained optimization formulation has been proposed that can accommodate both successful and failed demonstrations Shiarlis et al. (2016). Trajectory-ranked Reward Extrapolation (T-REX) Brown et al. (2019) can leverage sub-optimal demonstrations in IRL, making use of ranked demonstrations to learn a reward function. Later work improves upon T-REX by learning a policy via behavioral cloning on the demonstrations. They then automatically generate ranked trajectories by adding increasing amounts of noise to the learned policy Brown et al. (2020).

This paper assumes that we are in a reward-free paradigm, an MDP/R setting, where our goal is to learn a good policy while also learning a reward function from human feedback. We follow the standard reward learning framework that uses supervised learning to learn such a reward function, $\hat{r}_{\theta}$ Daniel et al. (2014); Christiano et al. (2017). In both scalar- and preference-based feedback settings, we consider segments of trajectories $\sigma$. Each segment consists of a sequence of states and actions {$s_{t},a_{t},s_{t+1},a_{t+1},...,s_{t+k},a_{t+k}$}.

SDP comprises two phases: (1) the reward model pre-training phase and (2) the agent update phase. In the reward model pre-training phase, we first gather a data set, $D_{\text{sub}}$, of $N$ sub-optimal state, action transitions. We then pseudo-label all transitions in $D_{\text{sub}}$ with rewards of $0$, resulting in $D_{\text{sub}}=\{s_{i},a_{i},0\}^{N}_{i=1}$. $D_{\text{sub}}$ is then used to optimize the reward model $\hat{r}_{\theta}$ with the mean squared loss in Equation 3 (see lines 2-4 in Algorithm 1). As a result, the reward model $\hat{r}_{\theta}$ becomes pessimistic because it learns to associate all sub-optimal transitions with a low reward. Without such a prior, the reward model initially has random estimates for the sub-optimal transitions. The only way to improve such estimates is to obtain teacher feedback. Therefore, by pseudo-labeling the sub-optimal transitions with $0$, we obtain a helpful reward bias before receiving any teacher feedback.

Next, in the agent update phase, we initialize the RL agent’s replay buffer $D_{\text{agent}}$ with $D_{\text{sub}}$ (see line 6 in Algorithm 1). The RL agent then briefly interacts with its environment and performs gradient updates according to its loss functions (see lines 7-12 in Algorithm 1). The agent update process changes the RL agent’s policy and generates new transitions, which are then stored in both the agent’s replay buffer $D_{\text{agent}}$ (see line 9 in Algorithm 1) and the reward model’s data set $D_{\text{RM}}$ (see line 10 in Algorithm 1). It is important to note that in standard scalar- and preference-based reward learning, we query the teacher for feedback on trajectory segments sampled from $D_{\text{RM}}$. Therefore, adding new transitions into $D_{\text{RM}}$ during the agent update phase is necessary to ensure that the teacher does not provide redundant feedback to the original sub-optimal transitions (as $D_{\text{RM}}$ was empty prior to the agent update phase). When it is time for the teacher to provide their first set of feedback (see line 13 in Algorithm 1), the feedback can cover a different region of the state and action space, relative to the original sub-optimal data. We empirically show that the agent update phase changes the RL agent’s policy by performing policy rollouts and analyzing the differences in state distributions (see Figure 5 in Appendix B).

One may immediately think to ask: is labeling sub-optimal transitions with an incorrect reward problematic? Using incorrect reward labels does induce some incorrect bias for both the reward model and the RL agent’s value network. However, as the transitions are sub-optimal, the bias for using an incorrect reward is low and does not greatly influence learning. Figure 6 in Appendix B shows that the majority of the reward values for sub-optimal transitions gathered through a random policy do have true rewards close to 0. In addition, by using the sub-optimal transitions, we increase the overall amount of data used by both models, which can decrease the models’ variance Yu et al. (2022).

For ease of obtaining sub-optimal data for SDP, we used state, action transitions from a random policy. We further note that we do not require explicit access to a sub-optimal policy; we only require state, action transitions from said policy. For all the experiments in Section 5.2, we used 50,000 transitions. Moreover, all reward model hyperparameters remained the same during both SDP phases (i.e., the reward model pre-train and agent update phases) as well as during the standard scalar- or preference-based reward learning that followed.

To demonstrate the generality and effectiveness of SDP, we apply SDP to (1) scalar-based and (2) preference-based HitL RL approaches. For the scalar-based experiments, we combine SDP with R-PEBBLE (a regression variant of PEBBLE Lee et al. (2021)). We compare SDP + R-PEBBLE against the following benchmarks: R-PEBBLE, Deep TAMER Warnell et al. (2018) (a scalar feedback HitL RL algorithm), and SAC. For the preference-based experiments, we further show the robustness of SDP by combining it with three SOTA preference-based algorithms. This results in the following benchmarks: PEBBLE Lee et al. (2021), RUNE Liang et al. (2022), SURF Park et al. (2022), and SAC. We treat SAC Haarnoja et al. (2018) as an oracle baseline because it learns while accessing the true reward function, which is unavailable to the other algorithms (SAC is the core RL algorithm used across all baselines). We include it as a baseline only to show an upper bound of task performance. See Appendix A for full implementation and hyperparameter details.

For evaluation, we show average offline performance (i.e., freeze the policy and evaluate it with no exploration) over five episodes using either the ground truth reward function (DMControl experiments) or the success rate (Meta-world experiments). We perform this evaluation every 10,000 training steps. To systemically evaluate performance, we use a scripted teacher that provides either a scalar rating of a single trajectory segment or preferences between two trajectory segments according to the true reward function. To thoroughly test the effectiveness of SDP, we perform evaluations on three robotic locomotion tasks from the DMControl Suite: Walker-walk, Cheetah-run, and Quadruped-walk, and six robotic manipulation tasks from Meta-world: Hammer, Door-unlock, Door-lock, Drawer-open, Door-open, and Window-open.

In our experiments, the results are averaged over five seeds with any shaded regions or error bars indicating 95% confidence intervals. To test for significant differences in final performance and learning efficiency (e.g., area under the curve, AUC), we perform Welch t-tests (equal variances not assumed) with a p-value of 0.05. See Appendix B, Tables 4, and 5 for a summary of final performance and AUC across all tasks.

To further understand the effectiveness of SDP, we perform further analysis of SDP across three dimensions: (1) the phases of SDP, (2) the amount of sub-optimal data, and (3) the number of feedback queries. To cover scalar- and preference-based HitL RL settings in both DMControl and Meta-world, we consider SDP + R-PEBBLE (i.e., scalar feedback) in Walker-walk and SDP + PEBBLE (i.e., preference feedback) in Cheetah-run and Door-open.