Assistance in Teleoperation of Redundant Robots through Predictive Joint Maneuvering

Motion candidate nodes are given a state describing the positions, velocities, and accelerations, of the robot’s joints, \(s_{t}\) , and a velocity command, \(u_{t}\) . Each individual node \(MC_{s_t, u_t}\) generates a set of possible joint motion candidates \(\mathbf {Q}_u = \lbrace \dot{\mathbf {q}}^{1}, \ldots , \dot{\mathbf {q}}^{m}\rbrace\) through sampling nullspace motion around the gradient of the Yoshikawa manipulability index [51]. PrediKCT uses the gradient of this index as a central point for searching possible nullspace motion as keeping this measure high during teleoperation can help avoid kinematic singularities that would constrain the system’s ability to follow arbitrary operator commands [37]. PrediKCT samples from a multivariate distribution with the mean defined as the average of the last chosen nullspace velocity vector and the gradient of the Yoshikawa manipulability index at the current position,where \(m(\mathbf {q}_{t})\) is the Yoshikawa manipulability index of the joint positions at time t and k is a scaling factor that sizes an identity covariance according to the distance between the last chosen nullspace velocity vector and the gradient of the manipulability index. This anchoring of updates to the sampling region keeps the controller around the gradient of the manipulability index while biasing sampling toward perturbations identified as promising in the last timestep.

A joint motion candidate leads to a new state \(s_{t+dt}\) through a forward transition model \(T(\cdot)\) that integrates the motion candidate over a fixed period of time dt. This new state is passed to a child user prediction node. Thus, a motion candidate node creates a set of \(|\mathbf {Q}|\) children, each of which is a user prediction node corresponding to a predicted future state from applying one of the joint motion candidates \(\dot{\mathbf {q}}^{i} \in \mathbf {Q}_u\) to the current joint states \(\mathbf {q} \in s_t\) : \(s_{t+dt} = T(s_t, \dot{\mathbf {q}}^{i}, dt)\) .

User prediction nodes \(UP_{s_t}\) are given a state \(s_t\) and generate velocity primitives representing possible operator velocity commands for that state. Each user prediction node utilizes an operator model that maintains a probability distribution over possible velocity commands for a given state: \(p(\cdot | s_t)\) . To create the next layer in the tree, a set of velocity commands \(\mathbf {U_s}\) is sampled according to this operator model: p( \(u_{t}^{j} \in \mathbf {U_s}) \sim p(u_{t}^{j} | s_t)\) . For each sampled velocity command, a new child motion candidate node \(MC_{s_t, u_{t}^{j}}\) is created using this command.

The root node in PrediKCT is a motion candidate node defined by the current state \(s_{0}\) and most recent operator velocity command \(u_{0}\) . User prediction nodes and motion candidates nodes are created in alternating layers as described above. The tree always ends with a layer of motion candidate leaf nodes representing the possible motion candidates for the final timestep of estimated operator commands. The height of the tree h and the time period dt over which motion candidates are applied together determine the time horizon that is considered by a given tree. Two branching factors affect the shape of the tree: \(|\mathbf {Q}|\) , the number of possible joint motion candidates generated per motion candidate node, and \(|\mathbf {U}|\) , the number of possible operator velocity commands considered per user prediction node.

The value calculation for nodes in PrediKCT begins with the leaf nodes and is calculated up to the root. PrediKCT makes use of a reward function over motion candidates, \(R_A(\cdot)\) , which can be designed to select desired types of motion. For example, \(R_A\) may produce higher reward for smooth motions, negative reward for moving toward joint configurations close to singularities, and large negative reward for self-collisions. A detailed example of an implementation for \(R_A(\cdot)\) is given as part of PrediKCS’ sample implementation in Section 4.1.3.

Since PrediKCT always ends with a layer of motion candidate nodes, calculating the value from the leaf nodes begins with evaluating a set of motion candidates. The value of a leaf motion candidate node \(MC_{s_{T}, \mathbf {u}_T}\) at the final timestep of T is defined according to the maximum reward value of its considered joint motion candidates,

For all user prediction node layers in the tree at any timestep t, the user prediction node \(UP_{s_{t}}\) uses the calculated value of each of its motion candidate children nodes to assess its expected value. User prediction node values are calculated according to a sum over all of their children nodes, weighted by the probability of an individual velocity command being chosen. A discount parameter, \(\gamma\) , where \(0 \le \gamma \le 1\) is also included in this value calculation. Setting this parameter to \(\gamma \lt 1\) prioritizes the reward given to nodes earlier in the tree and discounts the weight of node value calculations according to how far the nodes are from the current timestep,

Finally, non-leaf motion candidate nodes also consider the value assigned to their children user prediction nodes when determining node value,

All motion candidate layers internal to the tree use this value determination. Similarly, the root motion candidate node uses this same procedure to choose the optimal motion candidate, \(\dot{\mathbf {q}}^{*}\) ,

A continuous multi-armed bandit establishes a framework for optimizing sequential decisions about exploration of possible reward-generating actions. Ultimately, the goal of this exploration is to determine the action that leads to the highest expected future reward. This setting consists of a space of infinitely many “arms” that can be sampled at any given time to receive a sample reward out of a (unknown) probability distribution of rewards generated by that arm [2, 49]. After some exploration phase in which arms are sampled, the simple regret is the difference in expected reward from the optimal arm (the arm with the highest expected reward) and the current estimate of the optimal arm [8].

PrediKCS addresses the problem of redundancy resolution in teleoperation through the lens of a continuous multi-armed bandit where the arms represent a target joint configuration and the reward from choosing an arm is given by a function of the motion resulting from moving toward that target in the nullspace of the Jacobian. Then, minimizing the simple regret results in choosing the target configuration with the highest expected reward over predicted future motion.

More formally, the arms of the bandit correspond to possible configurations \(\mathbf {q}^*\) for the desired motion,where this desired motion is applied as possible in the nullspace at each timestep (here \(\alpha\) represents a gain parameter for this desired motion). For a given operator action commanding some task space motion, u, the final joint movement \(\dot{\mathbf {q}} = f(\mathbf {q}, \mathbf {u}, \mathbf {q}^*)\) is determined by

To select the optimal arm \(\mathbf {q}^*\) , PrediKCS searches for the configuration that maximizes the expected reward according to some arbitrary objective function \(R_A(\cdot)\) , which can be designed to incentivize desired types of motion. The example implementation of this reward function used for system evaluation is given in Section 4.1.3. Using a probabilistic model of operator commands, \(p(\mathbf {u} | s)\) , and a transition function consisting of a kinematic model of joint motion, \(s_{t+dt} = T(s_t, \dot{\mathbf {q}}, dt)\) , this creates a generative model that can be used to sample possible rewards obtained from different target configurations over some finite time horizon h.

Thus, PrediKCS frames the problem of selecting a target configuration as a multi-armed bandit over continuous joint space. Each “arm,” corresponding to different possible target configurations, results in some stochastic reward consisting of the expected value of using this configuration as a target for nullspace motion over a finite horizon of predicted future operator commands. This reward is stochastic due to the user model considering a distribution of possible operator commands at each timestep within this sequence.

The basic procedure of PrediKCS consists of using the time between operator commands to explore the expected rewards of bandit arms through generating sample rewards from these arms. These sample rewards are produced using a predictive model of the operator to analyze expected future motion. This process continues until a new operator command is received, at which time motion is generated using the target joint configuration for nullspace motion corresponding to the current best-performing bandit arm. The result is an anytime algorithm that explores the space of possible target joint configurations until an operator command is received. Once a command is received, the desired end effector motion is translated into joint motion with minimal latency using the existing estimate of the optimal bandit arm.

The key exploration action in multi-armed bandits consists of selecting arms to receive a sample from, providing information about the expected reward from picking that arm. To implement this process of arm selection, PrediKCS evaluates a given candidate target joint configuration, \(\mathbf {q}^c\) , through Monte Carlo estimation of the expected reward using rollouts of future joint motion and predicted user commands. These rollouts can be evaluated according to the desired reward function to produce a sample reward for selecting that arm. PrediKCS allows specification of parameters dt and h representing the granularity and total time range in rollout generation, respectively, where h is some multiple of the sampling rate dt. Then, rollouts of predicted joint motion are generated h seconds into the future by utilizing the probabilistic user model to sample \(\frac{h}{dt}\) possible future user commands.

Given some starting state, \(s_t\) , defining joint states for all joints in the robot, a rollout \(v^{q^c}_{t:t+h} = \lbrace (s_t, \mathbf {u}_t, \dot{\mathbf {q}}_t),\ldots , (s_{t + h}, \mathbf {u}_{t + h}, \dot{\mathbf {q}}_{t + h})\rbrace\) is created by iteratively sampling possible user actions, determining the joint movement using target configuration \(\mathbf {q}^c\) , and applying the kinematic motion model to obtain the next state,for all \(0 \le i \le \frac{h}{dt}\) , \(i \in \mathbb {Z}\) .

The sample reward for this rollout is given by the sum of time-discounted rewards,where \(\gamma\) is a discount factor, \(0 \lt \gamma \le 1\) .

While many bandit algorithms assume a stationary reward, this particular setting may involve shifting rewards. As the robot’s joint state changes over time, the expected reward from following a nullspace target may shift as well. For non-stationary bandit problems, one strategy extending across specific algorithms for regret minimization is to use a rolling window of arm sample rewards to estimate the value of that arm [9, 16, 47]. PrediKCS follows this approach to modify an algorithm developed in the stationary bandit setting to this non-stationary application of robotic motion.

To sufficiently search the joint space for \(\mathbf {q}^*\) while maintaining a responsive teleoperation control system, PrediKCS reuses arm rollouts from previous timestep over some rolling window of length \(\tau\) seconds. The choice of \(\tau\) balances between model accuracy and search thoroughness: A large value of \(\tau\) will generate a thorough search of the joint space for promising target configurations but will also involve estimating arm reward distributions using outdated user models and initial states. Thus, selecting a \(\tau\) value is an important part of system tuning to achieve informative and accurate knowledge of which arm should be selected at each timestep.

To reuse these rollouts after their generation time, rollouts are generated for \(h + \tau\) seconds. Then, for a rollout generated at time t, this rollout can be temporally aligned with rollouts generated at time \(t+\epsilon\) , \(0 \le \epsilon \le \tau\) by considering rollout steps \(\lceil \frac{\epsilon }{dt}\rceil\) through \(\lceil \frac{\epsilon }{dt}\rceil + h\) , denoted asFigure 1 illustrates this alignment of rollouts generated at differing times.

In practice, PrediKCS discards rollouts where the predicted state \(\mathbf {q}_{\lceil \frac{\epsilon }{dt}\rceil } \in v^{q^c}_{t+\lceil \frac{\epsilon }{dt}\rceil }\) is sufficiently far from the actual joint configuration \(\mathbf {q}_{t+\epsilon } \in s_{t + \epsilon }\) to only consider rollouts that have (sufficiently closely) modeled the correct current joint state at the temporally aligned rollout timestep.

With nullspace motion selection thus framed as a continuous multi-armed bandit problem, the problem reduces to estimating the bandit arm with the highest expected reward through minimizing the simple regret. To do this, PrediKCS uses a state-of-the-art algorithm for sequential exploration for function optimization, Voronoi Optimistic Optimization (VOO), to sample possible arms on which to perform reward-estimating rollouts. VOO is specifically developed for function optimization of high-dimensional search spaces [27], fittingly for this application of bandit arms corresponding to high-DOF joint configurations. VOO uses Voronoi partitioning, more densely sampling areas of the joint space that show more promising potential reward.

As developed by Kim et al. [27], VOO selects arms from the continuous joint space by iteratively sampling either randomly over the entire space or in a targeted manner within the Voronoi cell created by the current best-performing arm. More precisely, for each new sample to be rolled out, PrediKCS samples a random joint configuration with probability w, or with probability \(1 - w\) , it samples a joint configuration closer to the configuration with the highest expected reward than any other (where w is a system parameter specified such that \(w \in [0,1]\) ). In practice, the sampling step in VOO is often implemented by sampling from a Gaussian centered at the best performing configuration until finding a new configuration within this best configuration’s Voronoi cell [31]. PrediKCS follows this approach to sampling the Voronoi cell of the current best configuration.

While the original application for VOO assumes a deterministic transition function, this application has effectively a probabilistic state transition model due to the probabilistic model of future operator commands. To compensate for this, when a new sample is selected, Monte Carlo estimation is used to estimate this sample’s reward across k rollouts instead of evaluating based on a single rollout.

The full sampling algorithm, presented as an anytime algorithm that will continue to produce samples until TimeRemaining() returns false, is presented in Algorithm 1. An example of the resulting clustering behavior of the VOO-based sampling method is illustrated in Figure 2.

The example PrediKCS parameter implementation uses a similar reward function to PrediKCT. This sample reward function is a weighted combination of three terms: distance error, singularity avoidance, and joint limit avoidance.

Distance error ( \(R^1_A\) ) evaluates the distance between the resulting position from following the proposed joint motion candidate over time window dt, versus the position obtained by directly applying the velocity command to the end effector’s pose (given by the forward kinematics function operating on the state, \(fk(s_t)\) ) over that same time window. The singularity avoidance term ( \(R^2_A\) ) incentivizes joint configurations that increase the Yoshikawa manipulability index (using \(\mathbf {J}_{t+dt} = \mathbf {J}(\mathbf {q}_t + dt \dot{\mathbf {q}})\) , where \(\mathbf {q}_t \in s_t\) ), while the joint limit avoidance term ( \(R^3_A\) ) counts the number of joints within some \(\epsilon\) of their limit (if the joint is non-continuous),

The final reward is a weighted combination of these terms:In the example implementations evaluated through a user study, the tuned set of weights used is \(\lbrace w_1 = -50.0, w_2 = 5.0, w_3 = -0.5\rbrace\) , which keeps the order of magnitude roughly the same between reward functions. Note that \(R^2_A(\cdot)\) has a positive weight, since higher manipulability indices are desirable while the other two functions represent objectives to be minimized.

Given a discrete set of goals, \(\Phi\) , this operator models calculates the likelihood of an individual goal given a particular state and action as proportional to the expected reward for that goal. We define the expected reward for a given goal as the expected decrease in distance to that goal, \(R_{H, \phi }(s, \mathbf {u}) = dist(s, \phi) - dist(s, T(s, \mathbf {u}, dt))\) , where \(T(s, \mathbf {u}, dt)\) models the expected resulting state from applying action \(\mathbf {u}\) in state s over time dt. This likelihood is converted into the estimated probability of each goal using a softmax over the goal set,

The operator model obtains the likelihood of each goal, \(l(\phi)\) , according to the combined probability over a trajectory of state–action pairs \(\xi = \lbrace (s_0, \mathbf {u}_0), \ldots , (s_t, \mathbf {u}_t)\rbrace\) ,

Finally, the model creates a categorical distribution over the goal set by normalizing over the individual goal likelihoods,

To generate the set of rollouts \(V^q\) for configuration q from state \(s_t\) at time t, this model first uses inverse transform sampling to pick a possible user goal \(\phi ^*\) weighted by the current estimated probability distribution over possible goals, \(\phi ^* \sim p(\Phi |\xi _t)\) . Then, at each state in this rollout, a command toward the selected goal is sampled by first creating a twist, \(\mathbf {u}^*\) , in the direction of \(\phi ^*\) from s. The sampled command \(\mathbf {u}\) is created by drawing from a Gaussian for each dimension of the twist command centered at \(\mathbf {u}^*\) and with variance equal to some fraction of the magnitude of the twist: \(\mathbf {u} \sim \mathcal {N}(\mathbf {u}^*, |\frac{\mathbf {u}^*}{c}|)\) (in practice, a variance \(\frac{1}{10}\) the magnitude of the ideal twist generates appropriately varied commands). Finally, \(\mathbf {u}\) is scaled according to the magnitude of the last recorded operator command, \(\bar{\mathbf {u}} = \frac{\mathbf {u}}{||\mathbf {u}||}||\mathbf {u}_t||\) .

Since PrediKCS utilizes a rolling window of samples spread out over \(\tau\) seconds, samples to be considered at the current timestep may be anywhere between 0 and \(\tau\) seconds old. As the operator model shifts over time, this results in old configuration rollouts having been sampled under a different probability distribution than the currently estimated probability distribution of the operator’s goal. Consequently, PrediKCS uses importance sampling to weight rollouts at old timesteps according to the probability distribution of the goal estimation at the current timestep.

More formally, consider the estimation of expected reward at timestep \(t + \epsilon\) from some target null-space configuration \(\mathbf {q}^c\) that was originally sampled at time t, where \(0 \le \epsilon \le \tau\) . The rollouts for this configuration were generated using the probability distribution over goals \(p(\Phi |\xi _{t})\) , while the current operator model gives the updated distribution \(p(\Phi |\xi _{t + \epsilon })\) . For a particular rollout \(v^{q^c}\) , let the mapping \(\phi _{v^{q^c}}\) give the sampled goal toward which actions were generated for this rollout. Then, to estimate the reward for configuration \(q^c\) using the rollout set \(V^{q^c}\) , importance sampling weights each rollout by the sampling ratio of this goal under the current and former probability distributions,

With each of these four controllers, participants attempted to complete two tasks through controlling the arm of a Fetch robot using a PlayStation 4 controller. Participant inputs on this controller mapped to end effector twist commands, with particular button presses operating the end effector’s gripper to manipulate objects in the environment. The magnitude of twist commands was continuous using the PlayStation 4 controller’s dual joysticks to control the three axes (one joystick controlled the Z axis and the other controlled X and Y axes), while participants selected between linear and angular control modes with the bumper buttons on the back of the controller. Linear velocity commands were capped at 0.075 meters per second per axis while angular velocity commands were capped at 0.2 radians per second. While the Min Vel, Short-Horizon PrediKCS, and Long-Horizon PrediKCS controllers can all turn user commands into joint motion in only the time it takes to calculate the Jacobian pseudoinverse at the current joint configuration, the RIK Target controller demonstrated smoother motion with a slower control update that allowed time for reaching the target joint velocities after the IK solution was generated. Consequently, all controllers were set to a control update rate of 10 Hz, which we found through pre-testing to create generally smooth motion from each controller. The experimental tasks and control setup were designed to represent the operation of consumer-grade robots. While teleoperation of fixed-base, safety-critical systems such as surgical robots usually involves precise control rigs with high-frequency input, control of mobile-base, consumer-grade robotic systems is often done through video game controllers for the benefit of user familiarity. For example, video game controllers have been used for teleoperation of mobile manipulators in data collection at scale [6] and even bomb disposal [17, 38].

The first task consisted of picking up a cube on the ground in front of the robot, then placing it in a bin on the ground beside a robot. This task used a pick-and-place style of motion, with the end effector remaining pointed toward the ground in the shortest possible path. The second task involved grabbing a plant from a covered container on one side of the robot and placing it upright in another bin on the other side of the robot. This task used a grasp-and-carry type movement, with the shortest possible path keeping the end effector horizontal throughout the movement but rotating the orientation within the plane. The general experiment setup is shown in Figure 3.

Before the experiment began, participants were given instructions on how to operate the robot using the PS4 controller and given 2 minutes to get familiar with the controls through open-ended practice. During this control period, participants used the Min Vel controller. After this, the experimented explained the two tasks to be completed to participants before beginning with each controller in the designated order. Participants were not told details of the four controllers before operation but were simply told that the four controllers would affect how their inputs were transformed into joint motion.

Participants were given a max of two attempts per task for each controller. A task was classified as a failure if the task was rendered impossible during operation (e.g., from knocking an object to be grasped out of the robot’s reach) or if participants took more than 5 minutes on one portion of the task. In our analysis, we classify a single failure followed by a success as an incidental error and two consecutive failures as evidence of inability to accomplish the task. Consequently, if participants completed the task in either of the two attempts, then only the successful task completion trajectory was used for analysis. Otherwise, if both attempts with a single controller resulted in failure, then the two failed trajectories were concatenated for analysis. The experiment took approximately 30–60 minutes depending on participant control ability, and participants were compensated $10 for their participation.

A total of 13 participants completed the experiment. One participant’s data were discarded for not following task instructions, providing data from 12 participants (8 male and 4 female) divided into three balanced Latin squares. On a five-point Likert-style questionnaire, participants reported medium familiarity with robots ( \(M = 2.67\) , \(SD = 1.37\) ) and medium-to-high familiarity with video games ( \(M = 3.50\) , \(SD = 1.19\) ).

The effect of these four controllers on the success of the participant’s control is illustrated through analyzing both the local responses to participant commands and the entire trajectories of the task completions. During the experiment, robot joint states and participant motion commands were recorded at a rate of 20 Hz. The measures presented result from analysis of these recorded trajectories, combining trajectories between the two tasks, and ignoring timesteps at which the participants issued no commands and the robot was stationary.

The Path Distance metric gives the total distance traveled by the end effector while completing the tasks as calculated by the sum of pose distances between all consecutive end effector poses. This measure reflects how optimally the participant is able to complete the task with the given controller.

Next, the Joint Distance metric gives the total distance in radians between joint configurations across a trajectory, summed over all seven joints. This metric illustrates how much the joints of the arm actually moved across trajectory completion.

The Local Error metric gives the sum of squared pose distances over a trajectory between the actual end effector pose and the ideal pose—that is, the end effector pose that would result from applying the participant’s last twist command directly to the previous recorded end effector pose over the time between these two data points. This measure demonstrates how precisely the robot’s end effector motion follows the participant’s issued commands, rising when factors such as joint limits, joint dynamics, or kinematic singularities inhibit operator commands from being followed precisely.

Finally, the High Error Actions metric represents the number of participant commands that resulted in extremely high errors between ideal pose and actual pose. To calculate this number, the local errors for all actions across all participants and all controllers are first computed. Then, the distance value of the 95th percentile of these total errors is used as the threshold for these actions. That is, the number of High Error Actions in a trajectory is the number of actions in this trajectory that fall into the worst \(5\%\) of actions across all data points.

To calculate pose distances in these results, the linear distance in meters l and angular distance in radians a (calculated as the geodesic distance between two orientation quaternions) were combined through a weighted blending,where \(w_1 = 2.67\) and \(w_2 = 1.0\) . This weighting adjusts for the participant’s command limits scaling of radians per second being 2.67 times higher in angular control mode than the meters per second scale in linear control mode.

The ANOVA illustrates a significant effect of controller type on Path Distance, F(3, \(33) = 8.36\) , \(p \lt 0.001\) . Post hoc comparisons using Tukey’s HSD test show significantly shorter paths for Long-Horizon PrediKCS ( \(M = 44.1\) , \(SD = 9.16\) ) than the Min Vel controller ( \(M = 65.9\) , \(SD = 18.3\) ), \(p \lt 0.001\) , and the RIK Target controller ( \(M = 58.5\) , \(SD = 16.2\) ), \(p = 0.021\) . The Short-Horizon PrediKCS controller ( \(M = 46.6\) , \(SD = 12.9\) ) also has significantly shorter paths on average than the Min Vel controller only, \(p \lt 0.001\) . No other pairwise comparisons are significant. Thus, both PrediKCS controllers allow operators to more directly accomplish the tasks than by using the Min Vel controller, although only the Long-Horizon PrediKCS controller with long-term prediction significantly outperforms the RIK Target controller at this metric.

On the Joint Distance metric, however, the analysis finds no significant effect of controller type, F(3, \(33) = 1.95\) , \(p = 0.141\) . Juxtaposed with the Path Distance results, this paints an interesting picture: Despite the Long-Horizon PrediKCS controller allowing operators to use significantly shorter end effector paths than the Min Vel and RIK Target controllers, this difference does not show up in the actual joint movement. This illustrates how PrediKCS works: Through “extra” nullspace motion, future end effector maneuverability is preserved at the cost of larger joint motion locally per timestep.

For Local Error, running an ANOVA shows a significant effect of controller type, F(3, \(33) = 5.83\) , \(p = 0.003\) . Using Tukey’s HSD shows that the Long-Horizon PrediKCS controller ( \(M = 0.365\) , \(SD = 0.231\) ) again has significantly lower error than both the Min Vel controller ( \(M = 0.705\) , \(SD = 0.295\) ), \(p = 0.021\) , and the RIK Target controller ( \(M = 0.741\) , \(SD = 0.495\) ), \(p = 0.008\) . Likewise, the Short-Horizon PrediKCS controller ( \(M = 0.384\) , \(SD = 0.218\) ) also significantly reduces Local Error from the Min Vel controller \(p = 0.034\) and the RIK Target controller \(p = 0.014\) . Again, there is no significant difference between the pairwise comparisons of the two PrediKCS controllers or the Min Vel and RIK Target controllers, respectively. These results demonstrate that the PrediKCS controllers give operators improved precision of motion commands on average over the other two controllers.

Finally, controller type is also found to have a significant effect on the number of High Error Actions per trajectory, F(3, \(33) = 6.44, p \lt 0.002\) . While Short-Horizon PrediKCS ( \(M = 168\) , \(SD = 112\) ) lessens the number of these actions from Min Vel ( \(M = 352\) , \(SD = 222\) ), \(p = 0.032\) , the Long-Horizon PrediKCS controller ( \(M = 108\) , \(SD = 64.9\) ) significantly decreases these actions over both the Min Vel controller ( \(p = 0.002\) ) and the RIK Target controller ( \(M = 333\) , \(SD = 260\) ), \(p = 0.005\) . No other pairwise comparisons show significant differences on Tukey’s HSD test. The PrediKCS control system, and particularly the use of long-horizon prediction in PrediKCS, reduces the amount of time spent in poorly maneuverable configurations where operator twist commands can not properly be converted into joint motion.

In general, these results show that long-horizon prediction through PrediKCS can lead to improved operator performance in teleoperation over both a baseline minimum joint velocity controller and a controller based on a state-of-the-art IK optimizer. While this version of PrediKCS may not lower overall joint movement, it can utilize this higher joint movement per timestep to allow the operator to more directly and accurately control the motion of the end effector in task space. PrediKCS using only local prediction illustrates improvements over the minimum joint velocity controller as well, though it only demonstrates significant performance improvements over the RelaxedIK-based controller on local error. Generally, the two PrediKCS controllers perform similarly, but the longer-horizon PrediKCS produces more consistently improved behavior over the other two controllers. Figures 4 and 5 illustrate these results.

To provide more insight into how Short- and Long-Horizon lowered error in this user study, Figure 6 illustrates the mean and \(95\%\) confidence intervals of the squared error in every \(5\%\) of the trajectory, separated by task (the two double failures were not included in this data). The visualization helps elucidate how error differs between the controllers over the trajectory: The Min Vel controller maintains generally larger error over most portions of both trajectories, while RIK Target appears prone to more numerous spikes in error than the PrediKCS controllers (perhaps due to occasional joint reconfigurations). The two PrediKCS controllers behave similarly with low error for much of the two trajectories along with occasional spikes. However, the two controllers differ in which portions of the two trajectories resulted in these spikes in error (e.g., Long-Horizon PrediKCS demonstrates higher error around the picking action at the midpoint of Task 1, while Short-Horizon PrediKCS gave users more difficulty in the placement section of Task 2).

This study did not illustrate significant differences in performance between a local PrediKCS system and a longer-horizon PrediKCS system. While both utilize prediction to some degree (and indeed some prediction is required for PrediKCS to generate samples ahead of time and run as an anytime algorithm), the longer-horizon PrediKCS might be expected to perform better than optimizing only locally. The data suggest that the longer-horizon PrediKCS was more consistent, demonstrated through its more consistent improvements over the other two controllers. One possible reason for this is that the operator model utilized only allows modeling of the next waypoint to which an operator is moving. This may have rendered long-term planning obsolete, since only considering local linear movement toward the current waypoint would produce similar joint movement to simulating the full movement toward this waypoint. A more complex operator model that could look ahead at future waypoints according to an inferred task may allow more complex joint motion planning to accommodate the full length of a task.

As Figure 6 suggests, advantages between shorter- and longer-horizon PrediKCS may also correlate with the layout of the specific task to be completed. We expect longer horizons to be better for tasks involving hard-to-reach poses for which early maneuvering may be necessary. Conversely, shorter horizons may perform better in the absence of these poses due to the more thorough exploration of the search over target configurations afforded by faster sample rollouts. Generally, the tunability of PrediKCS through adjustment of horizon lengths and operator models is notable. Future work on PrediKCS could further quantify the impacts of these parameters to determine how task specifics can inform this parameter tuning.

This study involved operators who were relatively novice at using each control system. Future work could investigate the adaption over time to each control system and whether participants might learn to trust a controller like PrediKCS as operation commands are more consistently translated into proper joint motion. Additionally, since these control systems require the input of twist commands, other control interfaces for twist commands could be evaluated. More natural (e.g., motion controllers) vs. more precise (e.g., joysticks with haptic feedback) control interfaces may produce different types of operator behavior; evaluating these control systems across further control interfaces would help elucidate the impact they may have across future interfaces for robotic control.

Furthermore, the discussed system implements one of several possible paradigms for teleoperation: task-space velocity control. Other pose-IK systems, such as RelaxedIK, implement task-space pose control. Finally, higher-level interfaces use semantically meaningful actions or partial trajectories from which users can select. A systematic method for determining which of these paradigms is appropriate for a given context is an open question. Future work investigating how different users react to different control paradigms could better inform when systems like PrediKCS are most appropriate and worth the computational expense.