Trustworthy Conceptual Explanations for
Neural Networks in Robot Decision-Making

Given a collection of input concepts for an action concept, we now derive a score to measure how well each input concept explains the action concept. If an autonomous vehicle inadvertently failed to yield to another vehicle (i.e., action concept), we can test whether the color or the type of the other vehicle (i.e., input concepts) influenced the decision. To formally define the score, let us decompose the neural network into two segments, $f_{-l}(\cdot)$ and $f_{+l}(\cdot)$, with one following the other, at the $l^{\text{text}}$ layer as, $\mathbf{y}=f(\mathbf{x})=f_{+l}(f_{-l}(\mathbf{x}))$ for input $\mathbf{x}$. In other words, $f_{-l}(\cdot)$ are the activations of a DNN at the $l^{\mathrm{th}}$ layer, which are then fed to the rest of the network, $f_{+l}(\cdot)$, to obtain the output, $\mathbf{y}$.

Since our objective is to build a relationship between the activations and human understandable concept inputs for some behaviors (i.e., action concepts), as shown in the plot of Fig. 1, we work in the space of activations (i.e., the space of $f_{-l}(\cdot)$). The dimensionality of the activation space is equal to the number of nodes in the $l^{\text{th}}$ layer. We can obtain the sensitivity of neural network decisions, conditioned by $C_{A}$, w.r.t. layer activations as $\frac{\partial f^{C_{A}}(\mathbf{x})}{\partial f_{-l}(\mathbf{x})}$. If we want to measure this sensitivity along a particular direction in the activation space, we have to compute the directional derivative. If this direction is a random variable, $V$, with its realizations, $\mathbf{v}$, is given by the stochastic directional derivative,

If we build a relationships between $V$ and $C_{I}$, we measure the sensitivity of the neural network decisions to human understandable input concepts. Since $V$ is a random variable, it tells us about the uncertainty of this relationship. To build this relationship, let us estimate $\mathbf{v}$ using an external dataset containing inputs from $C_{I}$. To this end, similar to  [6], we collect a concept dataset: $X^{C_{I}}=\{(\mathbf{x}_{m}^{C_{I}},z^{+})\}_{m=1}^{M}$ with input concepts and a different set of inputs $X^{C_{I}^{-}}=\{(\mathbf{x}_{m}^{C_{I}^{-}},z^{-})\}_{m=1}^{M}$ with the positive and negative classes labelled as $z^{+}$ and $z^{-}$, respectively. The negative input concept, $C_{I}^{-}$, can be just another concept or a random collection of inputs, depending on what the user is interested in comparing. How these sets are selected will be described in the Experiments section. See Fig. . 1 for an illustration.

By computing $f_{-l}(\mathbf{x}_{m}^{C_{I}})$ and $f_{-l}(\mathbf{x}_{m}^{C_{I}^{-}})$, we obtain the activation values of the concept dataset. If we can obtain a linear separation between the two classes, then the concept is more unique in the activation space. However, as we will discuss in Section III-B, separation in a high dimensional activation is subjected to uncertainty, we consider a a probabilistic linear separation (i.e., we can draw many separation lines with different probabilities). By setting this probabilistic linear separation the same as $\mathbf{v}$, we obtain the relationship between $V$ and $C_{I}$. To estimate the vector distribution, $\mathbf{v}$, that separates the two classes in the activation space, we apply the Bayes theorem,

Given the positive and negative labels, the likelihood follows a Bernoulli distribution, $z|\mathbf{v}\sim\mathrm{Bernoulli}$. The prior weight distribution is considered to follow a normal distribution, $\mathbf{v}\sim\mathcal{N}$. However, because of the non-conjugate prior, the posterior distribution is not tractable [23, 27]. Hence, we resort to approximate Bayesian inference. Because the activation space of the neural network can be high dimensional, rather than using an MCMC technique, we use variational inference, where we minimize the KL divergence between the true posterior and an approximate posterior distribution, $q(\mathbf{v})$. However, since the true posterior is not known, instead of minimizing the KL divergence, we maximize an evidence lower bound (ELBO),

Following the locally linear approximation of the posterior in [28], we learn $q(\mathbf{v})$ in an expectation-maximization-style.

On a collection of test inputs, $X^{\text{test}}=\{\mathbf{x}_{p}^{\mathrm{test}}\}_{p=1}^{P}$, with relation to (1, we can now compute a score, $s^{C_{A},C_{I},C_{I}^{-}}$,

This score itself is a distribution. If we obtain samples from $q(\mathbf{v})$, each of them is a valid explanation. Since some samples are more probable than others, those with high probability are more likely concepts that explanations the decisions. The empirical mean and standard deviation of the score can be estimated easily. The mean score $\bar{s}$ is given by,

where $V^{C_{I}}=\{\mathbf{v}_{r}\}_{r=1}^{R}$ are $R$ samples taken from $q(\mathbf{v})$. The higher the score, the better the concept $C_{I}$ explains the test inputs $X^{\text{test}}$. Similarly, the empirical standard deviation reflects the epistemic nature [23] of explanations—how much the model knows that its explanation can be wrong.

To assess the trustworthiness of explanations, relying on 1) the held-out test accuracy from eq. (2) and 2) the epistemic uncertainty of the score in eq. (4), we consider (Fig. 2):
Case 1 (Off-base explanations): If the accuracy is low, the concepts are not good enough to delineate the CAVs, resulting in an inaccurate explanation. Formally, in such cases, $p^{C_{I}}\approx p^{C_{I}^{-}}$, where the distributions are defined as $X^{C_{I}}\sim p^{C_{I}}$ and $X^{C_{I}^{-}}\sim p^{C_{I}^{-}}$ for the positive and negative classes, respectively. Such explanations should not be trusted.
Case 2 (Imprecise explanations): If the accuracy is high but the uncertainty is also high, multiple explanations are possible. This can be due to, 1) the two supposedly opposite concepts are not sufficiently different enough to be delineated with a low uncertainty (i.e., some conflicting information) or, most likely, 2) the test samples lack diversity in the activation space. Unfortunately, Bayesian inference cannot differentiate lack of information from conflicting information [29].
Case 3 (Precise explanations): If the accuracy is high and the uncertainty is low, then the concepts we have chosen are good and the explainer is able to provide consistently good explanations. These explanations are highly trustworthy.

The thresholds for probability should be decided by the practitioners depending on how much risk they are willing to take. For instance, if an engineer is using BaTCAVe to debug a manipulator used in an assembly line, the threshould can be selected leniently as the stakes might be relatively low. In such cases, we can obtain more valid explanations. In contrast, if a legislative body is using BaTCAVe to approve a new autonomous vehicle, then the thresholds should be strict. If case 1 violates, we should try new concepts to obtain a better accuracy. If case 2 violates, we can still use explanations but they might not always be the best explanations. By obtaining the mean score, we can obtain an average explanation.

Setup: We used a JetBot with a NVIDIA Jetson Nano (Fig 3a). It uses a pretrained AlexNet [30] with the last layer replaced with two nodes. Using 175 “obstacle” and 175 “free” images, we fine-tune the DNN so that the robot can learn to avoid obstacles. We analyze how the decision of the network is affected by different data pre-processing and fine-tuning techniques for $C_{A}=\{\text{avoid obstacle}\}$ and $C_{I}=\{\text{orangeness, darkness, distance}\}$.

Results: With the objective of avoiding orange obstacles, we fine-tuned the full DNN with images of an orange box, shown in Fig. 3a, until we achieve a validation accuracy of 100%. We surveyed 20 engineering students who have at least 1 year of experience training DNNs to get their opinion on what the DNN has learned just by reading our description. We described them the architecture and showed fine-tuning images without telling them that our objective is avoiding orange obstacles. First, we asked them to describe what attributes the DNN should have learned and then we gave them a list of potential concepts to narrow down their choices. Many human speculated that the model will be fined-tuned to distinguish orange from the rest.

In contract, showcasing the BaTCAVe’s ability to provide true explanations that engineers might not even think of, as shown in Fig. 4(a), BaTCAVe revealed that the model has learned to distinguish dark from light objects (or the value or brightness in HSV scale). The orange box is merely a shade of the broader “dark” concept. Similar to C-deletion in XAI literature [20, 24, 25], we verified that the darkness is an important concept by gradually varying the brightness of the orange object, as shown in Fig. 4(d). Additionally, based on Fig. 4, BaTCAVe made the following explanations:

• •

  The score is proportional to the concept of “distance between the robot and obstacle,” verifying that the DNN has learned the distance.

• •

  When the final layer is fine-tuned (LFT) instead of the full DNN (FFT), the prominence of the orange concept becomes higher compared to the dark concept, which is what we originally intended, demonstrating how engineers can use concepts to debug robots.

• •

  As shown in Fig 4(b), the importance of the dark concept remains consistent across different data augmentation methods—adding color jitter (CJ) and/or image rotation (R)—for LFT while it varies for FFT.

Since XAI methods are hard to quantitatively benchmark, similar to other work, Fig. 4(c) qualitatively compares why BaTCAVe is a better choice than feature attribution methods for robot learning.

Setup: We use a Panda, a 6-DOF robotic arm with a gripper, to complete the three tasks shown in Fig 3(b,c,d) on RoboSuite [31]. The setup includes various proprioceptive sensors to monitor the arm’s movements and positions. By using the DNN shown in Fig 5 and the train using the ph low dim dataset from robomimic  [32], we developed an agent based on behavioral cloning [32]. Robot’s actions are $\Delta$ differences. Our objective is to explain which concepts of the inputs, $C_{I}$={object, EEF position measurement, EEF quaternion measurement,gripper open width}, is responsible when the robot is taking a set of actions defined by $C_{A}=\mathrm{top~{}75\%~{}of~{}each~{}}\Delta~{}\text{action}$.

Results: We obtain notably high BaTCAVe score with a low uncertainty, corroborating that accurate object information from proprioceptive sensors, unlike in vision-based settings, helps with precisely performing the task. Table I shows sample scores of $C_{A}$’s tested across different $C_{I}$’s. Further, an analysis of per time step explanations for the lifting cube task, depicted in Fig 5b, explains that only the object pose and EEF position measurements matters when the EEF is moving down.

Setup: This experiment replicates the setup and conditions of Experiment IV-B, but proprioceptive sensors are replaced by a camera and the object information is not given to the model. The architecture is shown in Fig 6.

Results: We evaluated $C_{I}$=(images={cube, gripper, table}, language={proper language commands, gibberish, verbs}). Interestingly, language concepts reveals that the verb in the sentence matters more than the rest of the sentence at the time it lifts the object.

Setup: To simulate autonomous driving, we trained an end-to-end deep reinforcement learning policy using proximal policy optimization (PPO) [33] to steer and throttle a vehicle on Donkey Simulator. The network is a CNN followed by an MLP, trained end-to-end.

Results: We analyzed how various input concepts affect steering decisions ($C_{A}=\text{steering angle}>2$). Fig. 8 shows how the input concept of “black shades,” which correlates with roads, explains steering decisions. Around $t=35$, the car points out of the road, and when it steers back we observe that its BaTCAVe for “black shades” increases around $t=41$, indicating that it was able to recover by focusing on the road. To improve the policy, it is important to identify what part of the neural network learns incorrect information. By applying BaTCAVe on the last CNN layer ($\bar{s}=0$) and last MLP layer ($\bar{s}=0.822$), we found that errors do not originate in the CNN, which implicitly acts as an image encoder. Therefore, instead of improving the encoder, the policy needs to be trained more.

Fig 11 shows the concepts which were used across different experiments. Random concepts were creating by randomly picking a unique color for each pixel value in the image. We evaluated the models with dark-light concept, orange-random concept, distance 30-random concept, distance 45-random concept, and distance 60-random concept sets.

Fig 13 and Fig 14 shows the results of general model, finetuned with all parameters and final layer parameters, respectively. They are compared over different concepts and augmentation techniques. We see that in both the cases models were not able to understand dark difference between dark and light concept.

Fig 15(a) shows the screenshot of the survey that was circulated. Fig 15(b) is the distribution of $C_{I}$ chosen by the participants (20). Fig 15(c) shows the word cloud representation of all the replies gotten by the participants.

Architecture: The model architecture, depicted in Fig 5(a), employs a low-dimensional observation modality that integrates multiple sensor inputs: gripper position (2-vector), end-effector position (3-vector), object characteristics (10-14 vector), and end-effector orientation (4-vector). The network outputs a 7-dimensional action vector which includes delta values for the robot’s movements: three for end-effector position (x, y, z), three for orientation (roll, pitch, yaw), and one for gripper force.

The model has 3 main components: Observation Encoding, Multi-Layer Perceptron(MLP), Observation Decoding.

1. 1.

   Observation Encoding: Prior to processing, observed states undergo encoding through an Observation Group Encoder. This encoder handles observation modalities, which are object information, robot end effector position and orientation, and gripper state. The encoded observations are concatenated into a single vector representation.

2. 2.

   MLP: The concatenated vector is then fed into an MLP comprising a single hidden layer with ReLU activation, facilitating the extraction of high-level features from the encoded observations. The output dimensionality of the MLP is 1024.

3. 3.

   Observation Decoding: Following feature extraction, the output of the MLP is decoded to produce the desired actions. This decoding process involves a linear transformation, mapping the MLP output to the action space. The actions consist of seven elements, corresponding to the changes in position (x, y, z) and orientation (roll, pitch, yaw) of the robot end effector, as well as the gripper state.

Training: We train the model using the ph low_dim dataset from robomimic [32] on task can,lift and sqare. The training process begins with data normalization to ensure consistent input scaling. The training loop spans 500 epochs, each consisting of 100 gradient steps. During each step, the model calculates the mean squared error (MSE) loss between predicted and ground truth actions. This loss is then used to update the model parameters using the Adam optimizer.

This model was trained across three different task:

1. 1.

   Pick Place Can(PPC): The primary goal of this task is for the robot to accurately pick up a can from one location using a gripping mechanism and then move it to a specified area to place it down. Fig 17 shows the state across different timesteps and Fig 18 shows BaTCAVe performed across multiple $C_{A}$ on PPC.

2. 2.

   Nut Assembly(NA): This task requires the robot to pick up a hollow object and accurately pick its handle and fit it in the object which fits the hollow area. Fig 19 shows the state across different timesteps and Fig 20 shows BaTCAVe performed across multiple $C_{A}$ on NA.

3. 3.

   Lift Cube(LC): In the LC task, the robot’s goal is to lift a cube from the table. Fig 21 shows the state across different timesteps and Fig 22 shows BaTCAVe performed across multiple $C_{A}$ on LC.

Table II shows BeTCAVe scores across all tasks and $C_{A}$’s.

Due to the nature of proprioceptive sensors not using image inputs, we do not define $C_{I}$ as images like traditional methods. We use the inputs vectors as $C_{I}$ the vector input of the perticular concept would be taken from the input and rest of the dimesnion would be randomly selected for example $C_{I}=^{\prime}object^{\prime}$ would be the vector input of object(size 10-14) + random vectors(size 9) to make up the total dimension 19. Random concept were random vectors of size 19-23.

For this experiment we test BaTCAVe on the final linear layer which is just after the MLP layer in the architecture Fig 5(a)

The model used in experiment 3 takes in both image and language input, We keep one input constant to test concepts of the other input. Random concepts were samples from ImageNet.

1. 1.

   Image concept: We choose concepts from the input image we blur out the rest of the concept in the input to represent a concept as shown in Fig 23. BaTCAVe shows high variance but consistent score across all $C_{A}$ when $C_{I}=^{\prime}Image^{\prime}$.

2. 2.

   Language concept: We use proper instructions, gibberish instructions and just verbs as language concepts. Table III shows the chosen 3 concepts for the eperiment. Fig 24 shows BaTCAVe tested across all $C_{A}$ in LC task with language used as $C_{I}$.

Architecture: The network outputs a 360-dimensional action vector via a final linear layer, mapping the robot’s projected movements over the next 36 timesteps. This output includes the first 108 nodes for end-effector position (x, y, z), the subsequent 108 for orientation (roll, pitch, yaw), and the final 36 for gripper.

The main components are Visual Encoder, Visual Narrower, Task ID Encoder, Controllers

1. 1.

   Visual Encoder: This part of the model uses a ResNet (Residual Network) architecture for visual encoding.

2. 2.

   Visual Narrower: A linear transformation that reduces the feature dimension from the output of the ResNet (512 features) down to a lower-dimensional space (256 features).

3. 3.

   Task ID Encoder(CLIP): Vision transformer for encoding task IDs based on both textual

4. 4.

   Controllers

   1. (a)

      XYZ Controller: Controls the position in 3D space.

   2. (b)

      RPY Controller: Controls rotation around the roll, pitch, and yaw axes.

   3. (c)

      Grip Controller: Manages the actions related to opening and closing a robotic gripper.

Training: The Model is trained based on a behavior cloning (BC) policy tailored to handle dual inputs: high-resolution images (224x224x3) from the camera and verbal instructions given to the robot. We train the model for 100,000 epochs on 300 demonstrations with a batch size of 64 using a Huber loss function and Adam optimizer.

Architecture: The policy model architecture comprises a sequence of convolutional layers followed by fully connected layers. The input to the model is an RGB image, representing the current state of the environment. After the convolutional layers, a flattening operation is applied, transforming the 3D feature maps into a 1D feature vector. This vector serves as the input to the fully connected layers. The flattened vector is fed into a dense (fully connected) layer comprising 512 units. This layer integrates the features extracted by the convolutional layers to form a high-level representation of the input.

Training: The policy network was trained on PPO algorithm using Stable Baselines3 library. The parameters set for training included a learning rate of 0.0003, a rollout of 2048 steps per update, and a batch size of 64. Discount factor (gamma) was set at 0.99 with a Generalized Advantage Estimation (GAE) lambda of 0.95, which helps in balancing bias and variance. The policy clipping range was set to 0.2, and advantages were normalized to stabilize the training. Additionally, the value function coefficient was set at 0.5, and the maximum gradient norm was capped at 0.5 to prevent exploding gradients.

As highlighted in section V as one of the limitations, when the concepts are small, they might not provide a stong enough signal to distinguish from a random concept class. For instance, Cone Concept B and Random Concept in Fig. 25 are largely similar.