RoboPack: Learning Tactile-Informed
Dynamics Models for Dense Packing

Simulators developed to model rigid and non-rigid bodies approximate real-world physics, often creating a significant sim-to-real gap [57, 17, 41]. To address this, we use a graph neural network (GNN)-based dynamics model trained directly on real-world robot interaction data, aligning with data-driven approaches for learning physical dynamics [42, 36]. Recent works have demonstrated inspiring results in learning the complex dynamics of objects such as clothes [34], ropes [5], and fluid [26], with various representations including low-dimensional parameterized shapes [38], keypoints [30], latent vectors [24], and neural radiance fields [31]. RoboPack, inspired by previous works [29, 47, 2], focuses on the structural modeling of objects with minimal assumptions about underlying physics. This approach overcomes the limitations of physics simulators by directly learning from real-world dynamics. Prior work on GNN-based dynamics learning [48, 49, 50, 55, 6] heavily relies on visual observations for predicting object dynamics, failing to capture unobserved latent variables that affect real-world dynamics, such as object physical properties. To address this challenge, our method incorporates tactile sensing into dynamics learning and leverages history information for state estimation, offering a robust solution to overcome the constraints of vision-only models.

Reinforcement learning (RL) aims to derive policies directly from interactions. Our method contrasts with model-free RL approaches [40, 32, 12, 19, 27], by incorporating an explicit dynamics model, enhancing interpretability and including structured priors for improved generalization. Our work is closer to model-based RL [16, 13, 42, 46, 39, 62] in that we combine learned world models with planning via trajectory optimization. In particular, we learn world models in an offline manner from pre-collected interaction data, avoiding risky trial-and-error interactions in the real world. However, our approach is different from existing offline model-based RL [45, 59, 9, 54, 15] as it leverages multiple sensing modalities, i.e., tactile and visual perception. This multi-modal approach provides a more comprehensive understanding of both global geometry and the intricate local physical interactions between the robot gripper and objects. Moreover, our method addresses challenges in scenarios where visual observations are not always available. It uses tactile observation histories to estimate partially observable states, enabling online adaptation to different dynamics. This integration of offline model learning, multi-modal perception, and online adaptation equips our system with adaptive control behaviors for complex tasks.

Tactile sensing plays an important role in both human and robot perception [7]. Among all categories of tactile sensors, vision-based sensors such as [60, 8, 25, 33] can achieve accurate 3D shape perception of their sensing surfaces. In our work, we use the Soft-Bubble tactile sensor [22] which offers a unique combination of compliance, lightweight design, robustness to continuous contact, and the ability to capture detailed geometric features through high-resolution depth images [22, 52]. Previous studies have successfully integrated vision and tactile feedback in robotic manipulation using parallel grippers [4, 10, 28] and dexterous hands [44, 53, 61]. In these tasks, vision effectively offers a comprehensive understanding of the scene’s semantics, while tactile sensing delivers accurate geometry estimation for objects in contact that are often occluded. In our study, we explore the potential of integrating vision and tactile feedback for learning dynamics in tasks involving rich contact, occlusions, and a diverse set of objects with unknown physical properties, such as box pushing and dense packing.

The objective of RoboPack is to manipulate objects with unknown physical properties in environments with heavy occlusions like dense packing. To formulate this problem, we define the observation space as $\mathcal{O}$, the state space as $\mathcal{S}$, and the action space as $\mathcal{A}$. Our goal is to learn a state estimator $g$ that maps $\mathcal{O}$ to $\mathcal{S}$ and a transition function $\mathcal{T}$: $\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$.

To efficiently learn dynamics from real-world multi-object interaction data, we would like to extract lower-dimensional representations of observations like keypoints. Furthermore, we require a mechanism to fuse tactile interaction histories into these representations without full tactile future prediction. Finally, to solve real robotic tasks, we need to leverage our learned model to plan robot actions.

Thus, our system has four main components: perception, state estimation, dynamics prediction, and model-predictive control, discussed in Section III-B, III-C, III-D, and III-E respectively. They are used together in the following way:

First, the perception system extracts particles from the scene as a visual representation $o^{vis}$ and encodes tactile readings into latent embeddings $o^{tact}$ attached to those particles.

Secondly, the state estimator $g$ infers object states $s$ from any prior interactions, which includes a single visual frame $o^{vis}_{0}$, the subsequent tactile observations $o^{tact}_{0:t}$, and the corresponding robot actions $a_{1:t-1}$:

$\displaystyle\hat{s}_{t}=g(o^{vis}_{0},o^{tact}_{0:t},a_{1:t-1}).$

(1)

Thirdly, to enable model-predictive control, we learn a dynamics prediction model $f$ that predicts future states given the estimated current states and potential actions:

$\displaystyle\hat{s}_{t+1}=f(\hat{s_{t}},a_{t}).$

(2)

Lastly, the future predictions are used to evaluate and optimize the cost of sampled action plans. The objective is to find a sequence of actions $a_{0},...,a_{H-1}$ to minimize a cost function $\mathcal{J}$ between the final states and a given target state $s_{g}$:

$\displaystyle(a_{0},...,a_{H-1})=\operatorname*{arg\,min}_{a_{0},...,a_{H-1}\in\mathcal{A}{}}\mathcal{J}{}(\mathcal{T}(s_{0},(a_{0},..,a_{H-1})),s_{g}).$

(3)

The robot executes the best actions and receives tactile feedback from the environment, with which it updates its estimates about object properties.

In real-world robotic manipulation, visual observations are not always available due to occlusion, but knowledge about object dynamics requires interactive feedback. In this work, we leverage tactile feedback to help estimate world states.

History information is often used to estimate the current state in POMDPs [1, 18, 23, 51]. Similarly, we seek to incorporate tactile history information into state estimation by employing a combination of graph neural networks (GNNs) and long-short term memory (LSTM), as shown in Figure 3(a). We define our state as a tuple of object particles and an object-level latent physics vector, which capture the geometry and physics properties of objects respectively. In the following paragraphs, we describe how our method performs state estimation using history information.

At time $0<t\leq T$, our state estimator $g$ infers all states for $t=1,...,T$ autoregressively. Given the estimated previous state $\hat{s}_{t-1}$ and the tactile feedback at the previous and the current state $o^{tact}_{t-1:t}$, we construct a graph $G_{t-1}=\langle V_{t-1},E_{t-1}\rangle$ with $V_{t-1}$ as vertices and $E_{t-1}$ as edges. For each node, $v_{i,t-1}=\langle x_{i,t-1},c^{o}_{i,t-1}\rangle$, where $x_{i,{t-1}}$ is the particle position $i$ at time $t-1$, and $c^{o}_{i,t-1}$ are particle attributes. The particle attributes contain (1) the previous and current tactile readings, $o^{tact}_{t-1:t}$, and (2) the latent physics vector of the object that the particle belongs to, $\xi_{\mathcal{M}{}_{i},t-1}$, where $\mathcal{M}_{i}$ is the object index corresponds to the $i$-th particle, $1\leq\mathcal{M}_{i}\leq Z$ and $Z$ is the maximum number of objects in the scene. Formally, $c^{o}_{i,t-1}=\langle\xi_{t-1},o^{tact}_{t-1:t}\rangle$. Note that here we implicitly assume that $\mathcal{M}$ is constant (i.e., objects only exhibit elastic and plastic deformations but not break apart), which generally holds for a large number of common manipulation tasks. Moreover, edges between pairs of particles are denoted $e_{k}=\langle u_{k},v_{k}\rangle$, where $u_{k}$ and $v_{k}$ are the receiver and sender particle indices respectively, and $1\leq u_{k},v_{k}\leq|V_{t-1}|$ where $k$ is the edge index. We construct graphs by connecting any nodes within a certain radius of each other.

Given the graph, we first use a node encoder $f^{enc}_{V}$ and an edge encoder $f^{enc}_{E}$ to obtain node and edge features, respectively:

$\displaystyle h^{v}_{i,t-1}$

$\displaystyle=f^{enc}_{V}(v_{i,t-1}),$

$\displaystyle h^{e}_{k,t-1}$

$\displaystyle=f^{enc}_{E}(e_{k,t-1}).$

(5)

Then, the features are propagated through the edges in multiple steps, during which node effects are processed by neighboring nodes through learned MLPs. We summarize this procedure as $f^{dec}_{E}$, which outputs an aggregated effect feature for each node called $\phi_{i}$:

$\displaystyle\phi_{i,t-1}$

$\displaystyle=f^{dec}_{E}(h^{v}_{i,t-1},\sum_{k\in\mathcal{N}_{i}}h^{e}_{k,t-1})_{k=1,...,|E_{t-1}|}.$

(6)

where $\mathcal{N}_{i}$ is a set of relations with particle $i$ as the receiver.

Next, the model predicts node (particle) positions and updates the latent physics vector:

$\displaystyle\hat{o}_{i,t}^{vis}$

$\displaystyle=f^{dec}_{V}\left(h^{v}_{i,t-1},\phi_{i,t-1}\right)_{i=1,...,|V_{t-1}|},$

(7)

$\displaystyle\xi_{\eta,t},m_{t}$

$\displaystyle=f^{dec}_{\xi}\left(\sum_{\begin{subarray}{c}i\\ \mathcal{M}_{i}=\eta\end{subarray}}h^{v}_{i,t-1},\sum_{\begin{subarray}{c}i\\ \mathcal{M}_{i}=\eta\end{subarray}}\phi_{i,t-1},m_{t-1}\right)_{\eta=1,...,Z}.$

(8)

where $f^{dec}_{\xi}$ is an LSTM, $m_{t}$ is its internal cell state at the current step, and $\xi_{\eta,t}$ is the updated physics latent vector for $\eta$-th object. At $t=0$ the LSTM state $m_{0}$ is initialized as zero. The physics vector for each object is initialized as Gaussian noise: $\xi_{\eta,0}\sim\mathcal{N}(0,0.1^{2})$ for all $\eta$. All other encoder and decoder functions (i.e., $f^{enc}_{V},f^{dec}_{V}$, $f^{enc}_{E}$, and $f^{dec}_{E}$) are MLPs.

After the state estimator produces an estimated state $\hat{s}_{T}=\langle\hat{o}_{T}^{vis},\xi_{T}\rangle$ from the $T$-step history, our dynamics model predicts into the future to evaluate potential action plans. The dynamics predictor $f$ is constructed similarly to the state estimator $g$, with two key differences: (i) it does not use tactile observations as input, and (ii) it is conditioned on frozen physics parameters estimated by $g$. Figure 3 illustrates this process. The forward prediction happens recursively: For a step $t>T$, we construct a graph in the same way as in Section III-C, but excluding tactile observations from the particle attributes, i.e., $c^{o}_{i,t}=\xi_{t}$. Then, the dynamics predictor infers the particle positions at the next step $\hat{o}_{t+1}^{vis}$ as formulated in Equations 5-7. The final state prediction is then $\hat{s}_{t+1}$ = $\langle\hat{o}_{t+1}^{vis},\xi_{t}\rangle$. Note that the estimated physics parameters are not modified by the dynamics predictor.

Training procedure and objective. We train the state estimator and dynamics predictor jointly end-to-end on trajectories of sequential interaction data containing observations and robot actions. For a training trajectory of length $H$, the state estimator estimates the first $T$ states, and the dynamics predictor predicts all remaining states. The estimation and prediction are all computed autoregressively. The loss is computed only on visual observations:

$\displaystyle\mathcal{L}{}=\frac{1}{H}\sum_{t=0}^{H-1}||\hat{o}_{t}^{vis}-o_{t}^{vis}||^{2}_{2}.$

(9)

Previous works [48, 50, 49] use the earth mover’s distance (EMD) or chamfer distance (CD) as the training loss, but these provide noisier gradients because EMD requires estimating point-to-point correspondence and CD is prone to outliers. Instead, we use mean squared error (MSE) as the objective, enabled by the point-to-point correspondences from our 3D point tracking (Section III-B). The details of the architecture and training procedure of the state estimator and dynamics predictor are in Appendix A-B.

Note that the learning of the latent physics information is not explicitly supervised. The model is allowed to identify any latent parameters that enhance its ability to accurately estimate the current state and predict future outcomes. We provide an analysis on the learned physics parameters in Section V.

With the learned state estimator and dynamics predictor, we perform planning toward a particular goal by optimizing a cost function on predicted states over potential future actions. Concretely, we use Model Predictive Path Integral (MPPI) to perform this optimization [58].

Planning begins with sampling actions from an initial distribution performing forward prediction with the dynamics models. The cost is then computed on predicted states. Based on the estimated costs, we re-weight the action samples by importance sampling and update the distribution parameters. The process repeats for multiple interactions and we select the optimal execution plan.

For computational efficiency, we execute the first $K$ planning steps. While executing the actions, the robot records its tactile readings. After execution, it performs state estimation with the history of observations and re-plans for the next execution. More implementation details on planning can be found in Appendix C.

To summarize this section, a diagram of the entire system workflow including training and test-time deployment is available in Figure 10.

We set up our system on a Franka Emika Panda 7-DoF robotic arm. We use four Intel RealSense D415 cameras surrounding the robot and a pair of Soft-Bubble sensors for tactile feedback. We use 3D-printed connectors to attach the Soft-Bubble sensors to the robot. Each Soft-Bubble has a built-in RealSense D405 RGB-D camera. The RGB data are post-processed with an optical flow computation to approximate the force distribution over the bubble surface [22]. Our hardware setup is depicted in Figure 4.

We demonstrate our method on two tasks where the robot needs to handle objects with unknown physical properties and significant visual occlusion: manipulating a box with an in-hand tool and dense packing.

To generate diverse and safe interaction behaviors, we use human teleoperation for data collection. In the Non-Prehensile Box Pushing task, for each weight configuration, we gather random interaction data for around 15 minutes. By “random", we refer to the absence of labels typically present in demonstration data. During these interactions, the end-effector approaches the box from various angles and contact locations, yielding diverse outcomes including translation and rotation, as well as relative movements between the in-hand object and the bubble gripper. The dataset contains approximately $12000$ total frames of interaction.

For dense packing, we collect approximately $20$ minutes of teleoperated random interaction data with five unique objects, randomizing the initial configurations of the objects at the beginning of each interaction episode. Each episode includes various attempts at packing an object into the box and includes pushing and deforming objects, as well as in-hand slipping of the in-hand object in some trials. The dataset contains approximately $6000$ total frames of interaction.

Though our dynamics model is orthogonal to the action space, suitable action abstractions are important for efficient planning and execution.

We compare our approach against three prior methods and baselines, including ablated versions of our model, previous work on dynamics learning, and a physics-based simulator:

1. i.

   RoboPack (no tactile): To study the effects of using tactile sensing in state estimation and dynamics prediction, we evaluate this ablation of our method, which zeroes out tactile input to the model.

2. ii.

   RoboCook + tactile: This approach differs from ours in that it treats the observations, i.e., visual and tactile observations $\langle o^{vis},o^{tact}\rangle$, directly as the state representation, whereas RoboPack assumes partial observability of the underlying state and performs explicit state estimation. This can be viewed as an adaptation of previous work [29, 48, 50, 49] to include an additional tactile observation component. With this baseline, we seek to study different state representations and our strategy of separating state estimation from dynamics prediction.

3. iii.

   Physics-based simulator: We also compare our method to using a physics-based simulator for dynamics prediction after performing system identification of explicit physical parameters. We use heuristics to convert observed point clouds into body positions and orientations in the 2D physics simulator Pymunk [3]. For system identification, we estimate the mass, center of gravity, and friction parameters from the initial and current visual observations with covariance matrix adaptation [14].

The considered methods, including our approach, share some conceptual components with prior offline model-based reinforcement learning (RL) methods (Section II-B), although with very different concrete instantiations. Each method either learns the full environment dynamics, or in the case of Physics-based simulator, performs system identification from a static dataset. All compared methods use the dynamics models to perform model-predictive control via sampling-based planning. Specifically, RoboPack (no tactile) can be framed as a model-based RL method (e.g., [59, 11, 9]) that uses only sparse visual observations for model learning. On the other hand, RoboCook + tactile treats visual and tactile observations as the state, overlooking the partially observable nature of the task. Our upcoming results demonstrate that our integration of multi-modal perception and physical parameter estimation leads to superior performance in challenging task domains.

Results are summarized in Table I. On the Non-Prehensile Box Pushing task, RoboPack is significantly better than alternative methods in all metrics. Compared to RoboPack (no tactile), RoboPack can better estimate the mass distribution of the boxes, which is crucial in predicting the translation and rotation accurately. In contrast, when using tactile and visual observations directly as the state representation (RoboCook + tactile), the performance is even worse than RoboPack without tactile information. We hypothesize that this is because the model has very high errors in learning to predict future tactile readings because of the intricate local interactions between the Soft-Bubble grippers and the object. The difficulty in learning to predict tactile reading may distract the model from learning to predict visual observations accurately.

Comparing RoboPack to a physics-based simulator baseline, we find that the simulator performs poorly on dynamics prediction for a few potential reasons, including (i) limited visual feedback for performing system ID, and (ii) the simulator’s parameter space may not capture the full range of real-world dynamics given the complex interactions between the compliant bubble and in-hand tool and rotating tool and the box. To illustrate the difference in model predictions, qualitative results are presented in Figure 6.

For the Dense Packing task, our model outperforms the best baseline on the pushing task, RoboPack (no tactile). We note that in this task, object movements are minimal and object deformation is the major source of particle motions. Metrics such as EMD and CD that emphasize global shape and distribution but are insensitive to subtle positional changes cannot differentiate the two methods in a statistically significant way. However, for the MSE loss, which measures prediction error for every point, RoboPack is significantly better than the baseline, indicating its ability to capture fine details of object deformation. This subtle performance difference between the two methods in dynamics prediction turns out to have a significant effect on real-world planning (Section V-D).

In this subsection, we seek to provide some quantitative and qualitative analyses of the latent representation learned by the state estimator. As it gives more direct control of object properties, we use our dataset collected for the Non-Prehensile Box Pushing task for the analysis.

To understand if the representation contains information about box types, we first attempt to train a linear classifier to test if there the features learned for different boxes are linearly separable in the latent space. We test the state estimator on $145$-step trajectories in the testing data, which typically involves three to five pushes on the box. The classification accuracy of physics parameters $\xi_{t}$ as more and more interaction information is processed is shown in Figure 7. It can be observed that as history information accumulates, the latent physics vectors become more indicative of the box type. In particular, the state estimator can extract considerable information in the first 20 steps, which is approximately the average number of steps it takes to complete an initial push. Furthermore, note that the state estimator only observes a history of no more than 25 steps during training, but it can generalize to sequences four times longer in this case.

To qualitatively inspect the learned representations, we perform principal component analysis, reducing the learned latent vectors from $\mathcal{R}^{16}$ to $\mathcal{R}^{2}$. Figure 7 shows the low-dimensional embeddings as the number of interaction time steps incorporated into the latents grows. We can see that as time progresses, the estimated latents become increasingly separated into clusters based on the physical properties (i.e., mass distributions in this case) of the manipulated object. The separation increases the most between $t=1$ and $t=20$, which is consistent with our observation in Figure 7 that longer histories than a certain threshold yield marginal returns.

Collectively, the results indicate that our state estimator indeed learns information related to physical properties based on interaction histories.

Next, we evaluate the performance of our approach in solving real-world robotic planning tasks.

For Non-Prehensile Box Pushing, we present quantitative results in Figure 9 and Table II. We can see that our method both achieves lower final error as measured by point-wise MSE (Table II) and makes progress toward goals more quickly (Figure 9) than other methods. The gap in performance between our model and RoboPack (no tactile) demonstrates the benefits of using tactile sensing in this task. While the physics-based simulator achieves the strongest performance of the baselines, it is not able to achieve as precise control as our method, taking more pushes to finish the task yet ending with higher MSE loss. We hypothesize this is because it can only infer dynamics of limited complexity via properties such as friction or mass center/moment. It also requires significant manual designs to construct the simulation for each task. Finally, RoboCook + tactile has the poorest control performance, consistent with its high dynamics prediction error on the test set. We hypothesize that the poor performance of this method is due to the difficulty of learning to predict future tactile observations, which are high-dimensional and sensitive to precise contact details.

For the Dense Packing task, we would ideally compare our method against the baseline with the best results on non-prehensile box pushing: the physics-based simulator. However, this is impractical for this task, because it is infeasible to obtain corresponding object models for the diverse and complex objects in this task or to estimate objects’ explicit physics parameters without visual feedback. Thus, we compare against the best among the remaining baselines instead, i.e., RoboPack (no tactile). We test on scenarios containing only training objects (Seen Objects) as well as scenarios where half or more of the objects are from the test set (Unseen Objects). Results on both settings, shown in Table III, indicate that our method is more effective in identifying objects that are deformable or pushable, which consequently enables the robot to insert the object at feasible locations. Examples of our experiments are illustrated in Figure 8. Despite our method having only seen rectangular boxes and plastic bags in the training set, it can generalize to objects with different visual appearances, geometries, and physical properties, such as the cups, cloth, and hat in the examples.