Corki: Enabling Real-time Embodied AI Robots via Algorithm-Architecture Co-Design

Detailed characterization data suggests the reasons contributing to slow execution of embodied AI robots are mainly twofold. First, the frame-by-frame sequential execution pipeline forces every action of the robot to undergo three stages: LLM inference, robot control and communication, and the latencies accumulate. Second, LLM inference, even with high-end GPU acceleration, is still extremely slow. The above reasons motivate this work.

From the perspective of a robotic system designer, the planning and decision making module does not need to match the high frequency of the control module. Trajectory is usually used as a bridge to eliminate the frequency mismatch. We use the same principle here.

We predict the continuous trajectory of the nearest future instead of discrete actions. We use a cubic function to fit the motion trajectory of robotic arms. For all the seven variables we need to predict, we output a trajectory for each one of the first six variables ($r_{x}(t),r_{y}(t),r_{z}(t),r_{\alpha}(t),r_{\beta}(t),r_{\gamma}(t)$), the gripper $g$ is still a binary value. Using $r_{x}(t)$ as an example, the model output will be shown as Equ. 4, $t$ is time.

After we change the model output, there are two ways of designing loss. The first one is directly supervising $a,b,c,d$. The second one is to supervise the trajectory with the ground truth. We go for the second one for two reasons. The first reason is that usually, no dataset provides the $a,b,c,d$ ground truth, which means we need to extract it from the trajectory ground truth first, which accumulates errors. Second, these parameters vary significantly and are not conducive to the learning of the neural network.

Using variable $r_{x}(t)$ as an example. We supervise the trajectory action $T_{x}$ in the training set and our predicted trajectory $r_{x}$ using the MSE shown in Equ. 5. Then, we update our trajectory parameters through backpropagation. In this way, we no longer need to get discrete actions with 30 Hz first and can directly monitor the trajectory to obtain a more capable model.

Because of our design, the robotic control and vision inputs are decomposed, leading to less information captured by the robots. To mimic this process during training, we intentionally insert fewer images. As shown in Fig. 4, vision-language tokens from $t=2$ to $t=4$ are shed by a mask embedding, similar to existing works such as BERT (devlin2019bert, ).

We thus provide flexibility in the length of the trajectory that is taken. The prediction length will be used as an upper bound of the length of the actual taken trajectory, and early termination is allowed. Again, assuming the prediction interval is 16.5 ms, the actual trajectory can be from 3.3 ms to 16.5 ms, with a stride of $\Delta t$ (which is 3.3 ms assuming the camera sensor works in a 30 Hz frequency). After the robot’s early termination, the model will predict another trajectory for the 16.5 ms.

Early termination gives us some flexibility, but it may not be enough. The reason is that the accuracy is higher when the actual trajectory length is consistent between training and inference. If the actual trajectory length is 6.6 ms in training, the same length should be taken during inference. Suppose the user wants to change the actual trajectory length. In that case, the only way is to train two models, one for 6.6 ms and one for 9.9 ms, and switch during inference, which is unsurprisingly inconvenient for almost all robotic applications.

Our method is to increase flexibility by allowing adaptive trajectory length with an empirical method. Our insight comes from the curvature of the trajectory. When the curvature is low, the action does not change significantly, suggesting a longer trajectory is acceptable. However, when the curvature is high, the usual circumstance is that the robot is encountering sudden change, where a shorter trajectory is better.

We identify the adaptive trajectory length using a concept called waypoints. For example, for a given trajectory spanning 16.5 ms, a waypoint is defined as a point on the trajectory every 3.3 ms or each time step. In Fig. 5, point $A$ is the starting point, points $B$ to $F$ are the waypoints, and point $F$ is the endpoint. Waypoint identification aims to find a waypoint where the robot’s movement is significant. In our case, significant movements are identified as high curvature or changes in the gripper state.

We check each waypoint from $B$ to $F$ and compare two metrics to identify potential waypoints with high curvature. Given the example in Fig. 5, the current waypoint undergoes checking is $D$. For every point in the interval of $[B,D)$, we compare two metrics with corresponding thresholds. The first is the $\angle{BAD}$ and $\angle{BDA}$ with a threshold of 90 degrees. The second one is the distance between point $B$ to line $AD$, or $d(B,AD)$ with a threshold $d$. If any threshold is violated, we consider the curvature at a point between $C$ and $D$ to be high, and thus, the trajectory should end at $D$ instead of the predicted point $F$. The length of the trajectory depends on the endpoint we get.

To find potential waypoints with gripper state changes, we compare the state of the gripper at the current waypoint and the next waypoint. If the gripper states of these two waypoints are different, the current waypoint will be identified as one with significant movement.

We explain the process in Algo. 1. As the adaptive trajectory length is determined during runtime, the latency is thus sensitive. The algorithm we propose is effective and with low latency. In most cases, the total computational cost of Algo. 1 is less than 500 FLOPs.

We provide users with an algorithm framework. Users can decide the length of trajectory prediction, whether early termination is needed, the level of early termination, and whether adaptive trajectory length is needed.

The task space computed torque control (TS-CTC) method is widely used in robotics for precise manipulation tasks due to its ability to handle reference inputs in the task space (murray2017mathematical, ). We show the control framework in Fig. 6.

The input of TS-CTC has two parts. The first part is the reference trajectory $x_{d}$, the first order derivative $\dot{x}_{d}$ (velocity) and the second order derivative $\ddot{x}_{d}$ (acceleration) of the reference trajectory. The second part is the joint angles $\theta$ and joint angular velocities $\dot{\theta}$ of the robot arm from the sensors. The output is the joint torque $\tau$. We describe the control process in Equ. 6. To achieve smooth robot control, the frequency of generating torques should be at least 100 Hz (yang2023dadu, ; dantec2021whole, ).

TS-CTC contains five key computing blocks, which are the most computationally intensive part of the whole process. We show them as red blocks in Fig. 6. The forward kinematics block calculates the pose $x$ of the end-effector in the task space based on the joint angles $\theta$. The Jacobian block calculates the Jacobian matrix $J(\theta)$ and the velocity $\dot{x}$ of the end-effector in the task space based on the joint angles $\theta$ and velocities $\dot{\theta}$. The task space mass matrix block computes the inertial matrix $M_{x}(\theta)$ of the robot arm in the task space based on the joint angles $\theta$. The task space bias force block computes the bias force $h_{x}(\theta,\dot{\theta})$ applied to the robot arm in the task space based on the joint angles $\theta$ and velocities $\dot{\theta}$.

We analyze the compute patterns of the above control algorithms and identify two key characteristics. First, as shown in Fig. 7, a significant amount of intermediate data is reusable. For instance, the calculation of the Jacobian matrix reuses results from forward kinematics. Similarly, the computation of the mass matrix and bias force reuses results from the Jacobian matrix and its transpose. Second, all blocks primarily consist of four basic operations: computing the pose of each link, the velocity of each link, the acceleration of each link, and the force of each link. Due to physical laws (e.g., acceleration is the derivative of velocity), these operations follow fixed data dependencies. For example, the velocity operator consumes a six-dimensional vector from the pose operator to calculate a six-dimensional vector representing velocity. A similar trend exists between the acceleration and force operators.

Leveraging the above analysis, our hardware design has two main goals. First, we aim to customize circuits and data pipelines to maximize intermediate data reuse, achieving high parallelization and performance. Second, we focus on customizing on-chip SRAM design to enable single read and write operations during computation, eliminating extra memory accesses.

Fig. 8 shows the Corki architecture, which consists of two parts. The blue blocks form a dataflow accelerator, where all main operators are connected through three FIFOs and a line buffer (LB). This design enables extreme pipelining; for example, the velocity calculation of the first link can start while the pose calculation of the second link begins. The yellow blocks are customized circuits, with the task space mass matrix unit reusing data from the pose unit and the task space bias force unit reusing data from both the velocity unit and the torque unit. There are occasional stalls in the accelerators due to differing latencies between the dataflow accelerator and the customized circuits. A simple micro-controller manages the control flow of the accelerator.

Our on-chip buffer design is highly effective. In the first four stages of the dataflow accelerator, three FIFOs store intermediate data, as the producer and consumer rates are identical. A line buffer between the force unit and the torque unit captures the rate mismatch between them. The remaining intermediate data is stored in a small scratchpad memory. This combination of different on-chip buffer designs allows for minimal on-chip SRAM consumption while ensuring no data communication with off-chip DRAM during execution.

We observe that robotic control has a unique feature: the compute frequency is high, yet the change in each control signal is low. For a 7-DoF robot arm, the movement in each joint is minimal each time. However, the computation of control signals is based on joints, as illustrated in the previous section. A joint-based approximation is possible to further save computation and reduce latency.

To quantitatively demonstrate our observation, we perform an experiment. We use a 7-DoF Franka Emika Panda robot arm (gaz2019dynamic, ) and monitor the item-wise changes in the mass matrix while slightly adjusting each joint by an angle. For example, we first record all the items in the mass matrix, then change the first joint by 0.1 radians (approximately 6 degrees), 0.3 radians (approximately 17 degrees), and 0.5 radians (approximately 29 degrees), monitoring the changes in the mass matrix. We repeat the same experiments for all the joints in the robot arm.

We show the results in Fig. 9. The results indicate that when motion occurs in joints 1 and 7, the mass matrix remains nearly constant. This phenomenon is illustrated in the top right and bottom right figures in Fig. 10. Movements in the end joints (joint 1 and joint 7) have minimal impact on the morphology of the robot arm, leading to less significant changes in the mass matrix. Similarly, for joints 5 and 6, the maximum variation in matrix elements does not exceed 0.1 even with an angular change of 29 degrees.

However, for the joints in the middle of the robot arm, the situation is different. When joint 2 moves, even a change of 6 degrees results in a maximum absolute change in matrix elements of 0.17 (with a maximum relative change of approximately 15.4%). When the motion increases to 29 degrees, the maximum relative change in elements can be as high as 45.2%. The bottom left figure in Fig. 10 shows that when the middle joints undergo movement, the morphology of the robot arm is significantly changed, necessitating the re-computation of all parameters in the control process.

We design a simple yet effective approximate computing method to dynamically update the control parameters, reducing the computational costs in the control process. Specifically, given the input $\theta$, we first compute the probability of each matrix (e.g., Jacobian matrix, mass matrix, etc.) needing an update based on an impact factor derived from the angular movement of each joint. In this process, the joints with a small impact on parameter changes have smaller impact factors, while the joints with a large impact on parameter changes have larger impact factors. The probability computation consumes less than 100 FLOPs, which does not affect the final latency.

If the probability of updating a matrix exceeds a certain threshold, the corresponding computation to generate that matrix is performed. Otherwise, the corresponding elements from the previous control cycle are reused. We observe that over 51% of matrix updates can be avoided without any loss in control accuracy.

The tasks are categorized into five types: moving an object, turning a switch on and off, pushing and pulling a drawer, rotating an object, and lifting an object. We use two metrics to evaluate the algorithm’s accuracy: success rate and average job length. The success rate is the most straightforward metric for quantifying a single task, calculated as the number of successful sequences divided by the total sequences. Given that the embodied AI algorithms are designed to improve robots’ abilities on long-horizon jobs, we further report the accuracy on finishing a job. Each job contains five consecutive tasks. The average job length measures how many tasks the robot can complete within a job, with a maximum of 5.

We further utilize two different metrics to illustrate why the results we predict are better:

• •

  Mean trajectory error. We compare the geographic distance between the predicted trajectory and the ground truth, using root mean square error (RMSE) as the metric. Generally, a smaller RMSE indicates better trajectory prediction.

• •

  Maximum trajectory distance. We also compare the maximum distance between the predicted and ground truth trajectories. A larger maximum distance denotes a higher likelihood of failure.

We evaluate the inference latency and energy consumption using a Nvidia V100 GPU, measuring power with NVML (vonnvml, ). Control latency and power are measured using an Intel 13th generation i7-13700 CPU. We implement Corki hardware on a Xilinx Zynq-7000 SoC ZC706 FPGA (zc706, ) to assess real hardware performance. Additionally, we establish Wi-Fi communication between a 7-DoF Franka Emika Panda robot arm (gaz2019dynamic, ) and our server to measure communication latency.

We train RoboFlamingo using the Calvin dataset for accuracy comparison. The results are either higher or equivalent to the reported version. For latency and energy consumption comparisons, we establish a baseline using the traditional execution pipeline of embodied AI algorithms, where the inference latency, control latency, and communication latency are accumulated each frame.

As discussed earlier, Corki can predict the trajectory of the next $N$ steps, with each step taking approximately 3.3 ms. Given the predicted trajectory covering $N$ steps, the robots can take anywhere from 1 step to up to $N$ steps. Longer steps reduce the inference frequency but may also lead to lower accuracy. In our evaluation, we predict nine steps each time and vary the steps taken from 1 to 9 with a stride of 2, creating five variations named Corki-T, where T represents the actual steps taken.

In addition to the fixed step variations, we evaluate adaptive options as discussed in Section 3.3. We name this variation Corki-ADAP. In Corki-ADAP, the robot’s steps are selected by the waypoints identification algorithm and are smaller than $N$.

We show accuracy results on seen scenarios and unseen scenarios in Tbl. LABEL:D-D and Tbl. LABEL:ABC-D. Almost all variations of Corki outperform the baseline in terms of both success rate and average job length, except for Corki-9 in unseen scenarios. On average, Corki improves the success rate by 8.6% and the average job length by 0.3. In unseen scenarios, these improvements are 8.1% and 0.2, respectively.

Among all fixed-step variations of Corki, Corki-5 achieves the highest accuracy and significantly outperforms the baseline. On seen tasks, it improves the average job length by 17.3% compared to the baseline, with a gain of 0.5 in job length. The trend observed among all Corki variations is that accuracy improves as the length of the actual trajectory taken increases. However, after reaching its peak accuracy, there is a gradual degradation in performance when the length of the actual trajectory taken continues to increase.

Corki-ADAP selects the length of the actual trajectory through identifying waypoints with significant movements. We observe that the results of Corki-ADAP fall between those of Corki-7 and Corki-5 in seen tasks, and it even outperforms Corki-5 in unseen tasks. This demonstrates that determining length during runtime is effective.

The improvement brought by Corki is significant. Corki outperforms the baseline in almost all cases because trajectory naturally provides more robotic-friendly supervision during algorithm training. When the datasets of embodied AI algorithms are constructed, the collection of the ground truth was in the form of trajectory at first. In contrast, if discrete actions with 30 Hz frequency are used for supervision, the trajectory must first be decomposed into actions in a frame-basis and then used to train the model. Second, a smooth trajectory with high frequency control certainly improves success rate, which are demonstrated in robotic community (kleff2021high, ).

When early termination of Corki is applied, the accuracy trend initially increases and then decreases. This is because the shorter the length of the actual trajectory, the closer it aligns with discrete action supervision. However, if the trajectory taken by the robot is too long, useful environmental information may not be captured and utilized effectively, as the closed-loop feedback also operates at a lower frequency.

Corki-ADAP works. This result validates our intuition that predicting a new trajectory whenever a significant movement occurs, such as a high curvature on the trajectory or a change in the status of the gripper, is beneficial.

The accuracy of our applications is directly related to the correctness of the trajectory. Therefore, we provide detailed trajectory data for evaluation. We compare the error on the trajectory and show it in Fig. 11. On average, Corki reduces the error by 25.0%.

However, we have also observed that a lower trajectory error does not always correlate with higher accuracy. For instance, although Corki-3 has a lower mean trajectory error compared to Corki-5, its success rate and average job length are lower. This discrepancy arises because the trajectory only reflects the trend of the robotic arm and cannot be treated as a perfect indicator of success rate. Additionally, this statistic does not account for the status of the gripper, which is also critical to the success of tasks.

We further illustrate the differences in trajectories with a real example. We compare trajectories on three dimensions separately and present the results in Fig. 12.

Although the baseline method can generate trajectories close to the ground truth on the Y dimension (Fig. 12b) and Z dimension (Fig. 12c), it clearly deviates from the target on the X dimension at time step 40 (Fig. 12a). In contrast, Corki maintains alignment with the ground truth across all three dimensions. These results again emphasize that while trajectory is related to the success rate, it cannot fully determine task success. Even though Corki’s trajectory slightly differs on the X dimension compared to the ground truth, it still successfully completes the task.

We compare the latency and present the results in Fig. 13 on the left y-axis. Corki significantly reduces the frame latency of embodied AI robotic applications. Among the variations, Corki-9 achieves the best speedup of $3.6\times$, as the inference frequency of the large language model is reduced by $8\times$. As the length of the actual trajectory taken increases from 1 to 9, the speedup gradually increases from $1.1\times$ to $3.6\times$. On the other hand, Corki-ADAP demonstrates a speedup of $3.0\times$, providing an ideal trade-off between accuracy and efficiency.

Corki also significantly saves energy consumption. Corki-1 has slightly higher energy consumption compared to the baseline, as it takes one step for every predicted trajectory, which is similar to the baseline. Besides Corki-1, all Corki variations have significantly lower energy consumption. Corki-9 has a $8.9\times$ energy reduction. Low energy consumption is critical to robots, which are mostly battery-supported devices.

We finally show frame-by-frame analysis of latency and energy consumption for one single sequence. Fig. 14a shows the results of latency and Fig. 14b shows the results for energy consumption. Both latency and energy consumption of Corki are having a same trend, where the crest indicates the inference of LLM is happening at that time step, and trough means the robot is executing the trajectory predicted from the last time. Corki-5 has a periodical crest, as every 5 time steps, the inference will happen once. Corki-ADAP has a more flexible crest and trough, compared to Corki-5. This is due to the waypoints identification and flexible length of actual trajectory.

The acceleration comes from three sides. First, the inference frequency is largely reduced, which contributes to the most latency reduction. Second, Corki hardware successfully accelerates the control process by up to $29.0\times$, reducing the control latency. Finally, communication latency between the robot and the server is hidden as we enable pipelining.