Complementary Learning for Real-World Model Failure Detection

Abstract

In real-world autonomous driving, deep learning models can experience performance degradation due to distributional shifts between the training data and the driving conditions encountered. As is typical in machine learning, it is difficult to acquire a large and potentially representative labeled test set to validate models in preparation for deployment in the wild. In this work, we introduce complementary learning, where we use learned characteristics from different training paradigms to detect model errors. We demonstrate our approach by learning semantic and predictive motion labels in point clouds in a supervised and self-supervised manner and detect and classify model discrepancies subsequently. We perform a large-scale qualitative analysis and present LidarCODA, the first dataset with labeled anomalies in lidar point clouds, for an extensive quantitative analysis.

I Introduction

In autonomous driving, analyzing anomalies in the environment surrounding the vehicle is a well-established research field [1, 2, 3, 4, 5, 6, 7]. However, many issues are not induced by rare events in the environment, but by model failures, which can also occur in seemingly normal situations. For such cases, Heidecker et al. introduced method-layer corner cases [3], and other data-driven perspectives followed [8, 9, 10]. Model failures in autonomous driving are rarely detected during evaluation, as labeled validation and test splits are typically very small and not representative. However, large-scale unlabeled fleet data recordings are often available.

In order to detect failure modes, there are many active research areas. Active learning [11] is concerned with continuously enriching the training data by querying samples from a set of unlabeled data points. Discrepancies between different sensor systems can also be used to query samples [12]. In error estimation, many approaches try to utilize unlabeled test sets for the evaluation of models [13]. Label refinement compares given labels, e.g. by an auto-labeling process, with new proposals [14]. All of these methods have in common that they utilize or compare two or more different results on the same task. However, we are unaware of approaches that leverage different training paradigms in order to utilize different data characteristics. Our main contributions are:

• Introduction and demonstration of complementary learning, leveraging complementary training paradigms to detect model failures

• LidarCODA, the first real-world anomaly dataset for autonomous driving with labeled lidar data

Figure 1: Model Failure Detection. The top left point cloud shows a supervised and the top right a self-supervised motion segmentation. The supervised model falsely classifies the pedestrian in the front left as static. Our approach exposes this model failure, as highlighted in red in the bottom image.

II Related Work

Figure 2: Method Overview. Given point clouds, we derive semantic motion labels in a supervised (sv) fashion. Complementary, we perform ground segmentation and derive predictive motion labels in a self-supervised (ssv) fashion. Subsequently, we perform point-wise discrepancy detection and classify potential model failures with an oracle.

The concept of comparing the outputs of two or more neural networks was already introduced in 1994 by Cohn et al., where they queried samples for active learning based on the disagreement between neural networks [15]. Since then, the variability in model predictions has been widely used to detect anomalies or errors. Ensemble diversity is especially well studied, as it was shown to lead to better performance [16], robustness [17], uncertainty quantification [18], and detection of outliers or distribution shifts [19, 20, 21]. While no uniform metric for ensemble diversity exists, measures like disagreement of models, double fault measure, or output correlation are widely used [22]. Ensemble diversity can be implicitly enhanced via random initialization [18], noise injection or dropout, or explicitly via bagging, boosting, or stacking. Compared to ensembles, mixtures of experts [23] include a learnable gate component for dynamic input routing. They enforce higher model specialization and thus more component diversity, leading to better detection of anomalous or out-of-distribution data [24].

The approaches mentioned above involve a combination of several neural networks with similar or identical architectures. Active learning is another research field interested in the detection of model failures. Here, uncertainty derived from ensembles is resource-intensive and thus only rarely used as part of a querying strategy [25, 26, 27]. Similarly, disagreements in a query-by-committee setting can be used to select samples [28]. In autonomous driving, also contradicting detections from sensors can be used as triggers, e.g., when radar and camera detections do not match [12]. Discrepancies between teacher and student models, typically known from knowledge distillation, can also be utilized [29]. As test sets are often small and not representative, directly estimating the accuracy of a model with only unlabeled data is of high interest [13, 30, 31, 32]. Here, we often see simple classification tasks or approaches that estimate an overall error that cannot be applied to individual samples. In some cases, generated pseudo-labels are utilized for further training steps [33, 34].

Disagreements can also be used for detecting erroneous labels. Ground truth labels in large vision datasets are often error-prone when auto-labeling processes based on large models are employed [35]. Detecting label errors with disagreements can be done by predicting a novel or refined label, and uncertainties can be generated by predicting multiple such labels [36, 14, 37]. This way, also noisy labels introduced by human errors can be detected [38].

Robustness during deployment is often achieved with sensor fusion, which, quite differently, purposefully aims to complement the weaknesses of one sensor with the strengths of another. Thus, disagreements are both typical and wanted, with the aim of resolving them [39]. However, also data from a single sensor can be split into multiple streams to increase robustness, e.g., by performing appearance- and geometric-based object detection [40]. In performance monitoring [41, 42], but also in outlier or anomaly detection [43], typically, a primary model performing a regular task is accompanied by a learned or model-based module that provides some sort of uncertainty for the results of the regular task.

Research Gap. Many of the analyzed works utilizing disagreements deal with toy problems and only analyze classification tasks, which are not sufficient to truly understand the shortcomings of a model that is designed for the complex task of autonomous driving. Many works analyze model outputs of the same architecture, leveraging differences during training. However, this way, the same data characteristics are being used during training. In autonomous driving, we see the largest potential for disagreement-based approaches in designing triggers for active learning [12] and increasing robustness during deployment [40], but these industry demonstrations are not accompanied by scientific works. Finally, to the best of our knowledge, no work exists that utilizes different training paradigms in order to detect model failures through disagreements.

III Method

We leverage complementary training paradigms on the same task in order to detect model failures and classify challenging scenarios with an oracle. The ability to detect model failures is based on the intuition that different training paradigms leverage different data characteristics from the same training data set. In this work, we demonstrate this approach with the motion segmentation of point clouds in the context of autonomous driving. As shown in Figure 2, we first derive motion labels from point clouds in a supervised and self-supervised fashion. Here, the first paradigm leverages human knowledge, given only context from static scenes. On the other hand, the second paradigm leverages temporal information inherent in the data. Typically, these paradigms are combined either in a pre-training concept [44] or with a combined loss during learning [45]. In our demonstrated use case, we examine motion labels in point clouds and focus on model disagreements. Based on a point-wise comparison, we detect discrepancies and cluster them for better interpretation. Finally, an oracle examines and classifies the model failures to better understand challenging situations. As one use case, our method, deployed within a query strategy in an active learning loop, can find exciting samples to improve a model under test. All code is on Github.

III-A Semantic Motion Labels

To derive semantic motion labels in point clouds, we first leverage a supervised semantic segmentation model [46] to determine whether a point belongs to a static or dynamic class. However, some classes do not provide clear information about the motion state of the points. For example, points assigned to the class bicyclist at a traffic light may be static in the case of a red light and dynamic in the case of a green light. We refer to such classes as potentially dynamic classes. By also performing supervised motion segmentation [47], we are able to further subdivide the existing classes. For example, the class bicyclist can be broken down into static bicyclist and dynamic bicyclist. Semantic classes that are static by definition remain unchanged. We refer to the semantic labels that are further subdivided based on their motion as semantic motion labels. An example of the resulting label fusion of semantic and motion labels to form semantic motion labels is shown in Figure 3.

Figure 3: Supervised Semantic Motion Labels. The left semantic segmentation allows no distinction between the parked car at the bottom left and the moving car at the top right. The middle image shows a motion segmentation, where the parked car was classified as static, and the moving car as dynamic. Finally, the right image shows the fused semantic motion labels. Adapted from [48].

III-B Predictive Motion Labels

In order to predict motion labels for a given point cloud, we first filter out the ground [49] to focus on objects in the scene, a common pre-processing step of scene flow models [50, 51, 52, 53, 54]. Based on self-supervised flow prediction [53] of the remaining points, we aim to derive motion labels, indicating whether a point is static or dynamic. The model takes consecutive point clouds as input and predicts the future motion for each lidar point in the form of a 3D displacement vector. The scene flow model does not distinguish between the point’s own motion and the observer’s ego-motion and represents the overall motion of a point between two consecutive frames. In order to derive relative displacements, we need to correct for the ego-motion. This can be done by leveraging or learning odometry information [55]. After predicting the future point cloud $\hat{X}_{t+1}=X_{t}+f_{t}$, we apply the learned rigid body transformation $T_{t+1\rightarrow t}$ of an odometry model, transforming the predicted point cloud back into the coordinate system of $X_{t}$. This gives the future point cloud $\tilde{X}_{t+1}$, which contains only the predicted relative motion without the ego-motion, as described by the following formula:

$$\tilde{X}_{t+1}=\left(T_{t+1\rightarrow t}\cdot\begin{bmatrix}(X_{t}+f_{t})^{T}\\ \mathbf{1}\end{bmatrix}\right)^{T}$$ (1)

As a result, static objects line up closely with the original data of $X_{t}$, and only dynamic objects show a predicted displacement, as shown in Figure 4 a).

Two-Stage Clustering. An analysis of the velocity values of the flow predictions showed that separating static from dynamic classes is infeasible in a point-wise fashion, as a strong overlap exists. What we found, however, is a significant difference when considering instance-wise normalized standard deviations, as shown in Figure 5. While we performed this analysis with ground-truth labels, in unlabeled data, the necessity arises to form instance clusters.

To obtain such instances, we utilize DBSCAN [56] to spatially cluster the point clouds, as shown in Figure 4 b). A cluster was classified as potentially dynamic if the normalized standard deviation of the cluster’s velocity was less than 0.12. This threshold was identified through a grid search, leading to the highest IoU for dynamic points using ground truth from SemanticKITTI [57]. In order to further reduce the number of false positives, we cluster potentially dynamic points based on their flow vectors in the second stage. This way, points that have a similar flow are clustered, causing fewer static clusters to be incorrectly classified as dynamic. Spatially separated points can now be grouped together based on their flow. An example of this step can be seen in Figure 4 c). Here, the points on the left and right edges belonging to static objects now form a cluster, which changes the distribution of flow vectors within this cluster, causing it to be classified as static. Finally, the newly found clusters were classified as dynamic if the speed of the cluster was above 4 km/h, based on typical velocity profiles of pedestrians, as visible in Figure 4 d).

Figure 4: Self-Supervised Predictive Motion Labels. The first image shows the original point cloud in green and the point cloud transformed by the scene flow model and compensated by the ego-motion using the odometry estimation in red. The second and third images show the result of the spatial and flow-based clustering, respectively. The fourth image shows the final predictive motion labels, with dynamic points in red and static points in green. Reprinted from [48].

III-C Discrepancy Detection and Failure Classification

After obtaining motion labels from both the supervised and the self-supervised stream, we are now interested in detecting contradictions between the labels, as shown in Figure 2. Given a semantic and a predictive motion label for each lidar point, there exist four different categories, where we assign a color for visual inspection to each. First, we color points both models deem static in green$\bullet$ and those both models deem dynamic in blue$\bullet$. For contradictory points, we color those red$\bullet$ where the supervised stream predicts a static point and the self-supervised a dynamic one, and all others yellow$\bullet$ in the opposite case. To help a subsequent human oracle better understand a scene, we provide a visual inspection tool where we map the lidar points onto the corresponding RGB image for an improved scene understanding, as shown in Figure 6. Finally, we cluster instances with contradicting labels, so a human oracle can classify complete scenes as well as single instances.

III-D Implementation Details

For all models shown in Figure 2, we utilized available architectures from the literature. We trained the supervised semantic segmentation model SalsaNext [46] and the supervised motion segmentation model of Chen et al. [47] on the KITTI-360 dataset [58], as it is a large dataset that contains semantic labels, motion labels, and odometry data. However, KITTI-360 only provides ground truth for accumulated point clouds and not for raw scans. To obtain ground truth labels for the raw scans, the labels for the raw point clouds were recovered with a nearest neighbor search. Here, we improved the work of Sanchez [59] in order to also recover labels of dynamic traffic participants. As a result, instead of one billion labels of the accumulated point clouds, about 6.9 billion labels of the raw scans were used for training. The training was performed on an NVIDIA RTX A6000. Hyperparameters were taken from the original papers [46, 47]. More details can be found in [48].

For the remaining three models, we used available pre-trained model weights. For ground segmentation, we deployed GndNet [49]. We chose FlowStep3D [53] as the self-supervised scene flow model because, at time of implementation, it had the lowest outlier rate among other self-supervised scene flow models on the KITTI scene flow dataset [60, 61]. The pre-trained model was trained on synthetic data of the FlyingThings3D [62] dataset with 8,192 points per point cloud. To minimize the domain shift mentioned by Baur et al. [63] between the training dataset with synthetic data and 8,192 points and the inference dataset with raw scans and about 120,000 points from a Velodyne HDL-64E scanner, the following preprocessing steps were necessary. First, only the lidar points from the field of view of the forward-looking camera were considered. Then, we removed points farther than 35 m and excluded points classified as ground. For the self-supervised odometry model, we deployed DeLORA [55].

Figure 5: Self-Supervised Label Generation. The left graph shows that the magnitude of point-wise flow vectors is not sufficient to distinguish between dynamic and static points. The boxplot on the right shows that the normalized standard deviation per instance is significantly lower for dynamic instances than for static instances. Ground truth from SemanticKITTI [57] was used to classify the points into dynamic and static. Reprinted from [48].

Figure 6: Supervised Model Failures. Based on our visual inspection tool, a human oracle can analyze contradicting model outputs and classify failure cases. In these exemplary images, we show model failures of the supervised stream, which can be detected due to contradicting outputs of the self-supervised model. Adapted from [48].

IV Evaluation

In the first section of the evaluation, we incorporated a human oracle for Model Failure Classification. In the second part of the evaluation, we analyze the sensitivity of the approach towards anomalies in the environment.

IV-A Model Failure Classification

We conduct an extensive analysis of our approach by examining the outputs of the models. Instead of analyzing only scenes that included detected discrepancies, we manually examined over 20,000 frames to better understand the model behaviors and scenes they react to.

Evaluation Dataset. To minimize perceptual failures due to a domain shift, it is important to choose an evaluation dataset similar to the training dataset. Both, different weather conditions and differences between datasets from different countries can lead to domain shifts [3]. However, classical evaluation splits are too small to truly understand the performance of our approach. Thus, we chose KITTI Odometry [58] for evaluation. KITTI-360, the dataset we trained on, and KITTI Odometry [64] are closely related datasets, as both were captured in Karlsruhe, Germany with a Velodyne HDL-64E lidar. Since we used the KITTI-360 dataset for training, and DeLORA was trained on sequences 00-08 of the KITTI odometry dataset, we used the remaining sequences 09-21 of the KITTI odometry dataset for the evaluation. We used sequences 09 and 10 to quantitatively examine motion segmentation performance for each part separately and used SemanticKITTI [57] labels as ground truth. To investigate model failures, sequences 11-21 of the KITTI Odometry datasets were used for qualitative analysis. As shown in Fig. 7, we observe that the majority of points are predicted as static by both models and around 5 % of the points show model contradictions. Due to the pre-processed point cloud of the scene flow model, the self-supervised stream predicts a label for significantly fewer points than the supervised stream, which predicts a label for all points in a lidar scan. In both analyses, only the lidar points per frame for which both streams predicted a label were considered.

Oracle. Sequences 11-21 of the KITTI odometry dataset were used for the qualitative analysis, comprising a total of 20,350 frames. Based on the visual inspection tool introduced in Section III-C, we present representative examples of detected model failures. In most cases, the models were correctly consistent, especially due to the high number of static points. In these areas, no improvement is necessary. Next, we observe model failures of the supervised stream, which is the model under test in most cases. Here, the two streams contradict each other, and we were able to detect the failure by the correct detection of the self-supervised stream. We show representative examples in Figure 6. Here, scene 1 shows a turning car and two moving bicyclists, where one bicyclist is wrongly labeled as static by the supervised stream. Scene 2 contains two walking pedestrians that are wrongly classified as static by the supervised stream. Scene 3 shows a parking car misclassified as dynamic by the supervised stream. Scenes 4 and 5 show a car moving slowly and a car moving backward, respectively. These rare cases also lead to model failures. These cases demonstrate effectively that our approach enables the detection of sometimes regular, but sometimes also rare and challenging scenarios that lead to model failures, which could not have been detected in a small test split. This way, new samples can be collected for labeling to further improve the model under test. We found various weak points in each stream, characterized by repeated occurrence. Specifically, the supervised model under test has weaknesses in distinguishing between dynamic and static objects in specific situations, e.g., at red lights or when a car is parked directly in front of the ego vehicle. An example of such situations is given in scenes 6 and 7 of Figure 6.

Figure 7: Discrepancy Detection. Given mostly normal data in the KITTI Odometry dataset, the two streams agree on 95 % of the data (green, blue). In light of anomalies, as provided by the CODA datasets, many more discrepancies (red, yellow) are detected.

Next, we show scenarios where the self-supervised stream showed model failures, detected by correct predictions of the supervised stream. Figure 8 shows representative scenes of this category. Scene 1 contains two distant pedestrians walking, wrongly classified as static by the self-supervised stream. In scene 2, a parked car is misclassified as dynamic by the self-supervised stream. Scenes 3, 4, and 5 show walking pedestrians or moving cars incorrectly classified as dynamic. These cases demonstrate that our approach enables the detection of challenging temporal scenarios. Often, models under test are fine-tuned variants of models trained in a self-supervised fashion. This way, additional training data can be collected for improved pre-training. Here, no labels are required. The self-supervised stream classifies an above-average number of objects as dynamic when the ego-vehicle turns or goes over a speed bump. An example is shown in Figure 8, where in scene 6, the vehicle turns, and in scene 8, it drives over a speed bump. Another weak point is fast oncoming vehicles on highways, which are often classified as static, as can be seen in scene 7. Finally, a common weakness is small clusters on the right or left edge that are incorrectly classified as dynamic, as in scene 9, where points of a window are classified as dynamic.

Finally, also cases occur where the models are incorrectly consistent. For this category, the two streams agree, but the label is incorrect in both cases. We show examples in Figure 10. Here, the left scene shows two walking pedestrians that are incorrectly classified as static and the right scene shows a parked car classified as dynamic.

Figure 8: Self-Supervised Model Failures. Based on our visual inspection tool, a human oracle can analyze contradicting model outputs and classify failure cases. In these exemplary images, we show model failures of the self-supervised model, which can be detected due to contradicting outputs of the supervised stream. Adapted from [48].

IV-B Sensitivity towards Anomalies

Generally, we are interested in scenarios where models fail. It is well known that perception models often struggle most with anomalies in the environment. Thus, we also examine the sensitivity of our approach towards object-level anomalies in the environment around the ego vehicle, as these are most often examined [65, 66].

Evaluation Data. While our goal in the first part of the evaluation was to minimize the domain gap between training and evaluation data, there are no labeled anomalies in the original KITTI dataset [64]. Thus, we utilized data from CODA [67], the only real-world dataset that provides lidar data and includes anomalies [68]. The CODA dataset provides anomaly labels for objects based on three existing data sets: KITTI [64], ONCE [69], and nuScenes [70].

Figure 9: LidarCODA. Annotated lidar scenes in the three data splits ONCE, KITTI, and nuScenes, from left to right. Anomalies are shown in red. Reprinted from [71].

For the CODA-KITTI split, the authors manually reviewed all misc labels available in the ground truth and relabeled some as anomalies according to a labeling policy. Since we trained our models on KITTI-360, this enables us to quantitatively examine our approach with only a small domain gap. For CODA-nuScenes, the authors similarly adopted available annotations in a manual process. Finally, for CODA-ONCE, they deployed an automated anomaly detection approach, making this subset the most relevant. CODA includes 1,500 scenes with a total of 5,937 anomaly instances. Among those, 4,746 belong to the superclass traffic_facility, followed by 929 vehicle and 197 obstruction instances. With 396, most vehicle instances can be found in CODA-KITTI. In Figure 7, we show the outputs of our approach on the CODA subsets, also in comparison to the outputs of the KITTI dataset. We can observe many more detected discrepancies, which is in line with the much higher number of anomalies, but the subsets reveal strongly varying behavior patterns.

For further qualitative analysis, however, labeled anomalies in 3D are required. The original CODA dataset provides anomaly labels only in the form of 2D bounding boxes in images. Therefore, we present LidarCODA, a dataset based on the CODA dataset [67] for evaluation. Based on a frustum-based filter, subsequent clustering, and manual inspection, we transferred the original 2D labels from image space into refined, point-wise 3D labels that go beyond the coarse characteristic of the provided bounding boxes. More details can be found in [71]. LidarCODA is the first real-world anomaly dataset with annotated lidar data, as shown in Figure 9. Here, also the different lidar systems utilized become clearly visible. Due to the sparse point cloud of nuScenes, many small or distant labeled anomalies in the image space are only covered by a few or no lidar points.

Dataset	#Frames	mIoU $\uparrow$	mP $\uparrow$	mR $\uparrow$	mF1 $\uparrow$
LidarCODA	1,412	8.9	13.2	26.2	17.5
LidarCODA-ONCE	1,034	8.9	14.0	27.1	18.4
LidarCODA-KITTI	307	10.9	13.3	29.0	18.3
LidarCODA-nuScenes	71	0.4	0.7	1.0	0.8

TABLE I: LidarCODA Subsets. Evaluation of our approach on LidarCODA and its three subsets. Adapted from [71].

Anomalies. With LidarCODA, we provide ground truth for object-level anomalies [1] in lidar data. Since our approach is not specific to object-level anomalies but shows its strength in underrepresented or generally challenging or atypical scenarios, which can also include scene- or scenario-level anomalies, we are primarily interested in the sensitivity of our approach towards these specific anomalies. This is different from works in anomaly detection, which aim to detect as many anomalies as possible. As our approach assigns distinct labels per point, we follow the evaluation protocols of semantic segmentation tasks rather than anomaly detection tasks, as here an anomaly score per point is required. First, we need to better understand the suitability of the subsets of LidarCODA due to introduced domain shifts, either due to new environments or due to new sensor setups. Table I shows the evaluation results on the individual subsets. Here, we evaluated all points of the lidar point cloud, even if our approach did not label individual points, e.g., because they were filtered during pre-processing. Such cases were counted as false negatives if an anomaly was missed. This way, we fairly incorporate the limits of our approach and do not create any requirements on the analyzed point clouds. The results clearly show that our approach struggles with the nuScenes subset, which is primarily due to the large domain shift w.r.t to the sensor setup. The subsets ONCE and KITTI, however, show more promising results, as here, our approach is capable of detecting anomalies. This allows for further investigation. As nuScenes is by far the smallest subset, its impact on the evaluation is rather limited.

Next, we investigate the sensitivity towards the superclasses provided by CODA. Here, we only evaluated points that have labels assigned by both our approach and the ground truth. As shown in Table II, our approach shows different levels of sensitivity given different types of anomalies. While animals were not detected at all, our approach is more sensitive to cyclists and objects of the misc class. The misc class consists of objects that are “unrecognizable or difficult to categorize” [67]. The results are difficult to interpret, however, as CODA defines an anomaly as an object that “blocks or is about to block a potential path of the selfdriving vehicle” [67] and/or “does not belong to any of the common classes of autonomous driving benchmarks” [67]. This risk-aware definition is not always in line with the methodology of our approach, where objects that block the path in front of the ego vehicle are not necessarily hard to segment or predict. Regardless, we can conclude that our methodology shows a heightened sensitivity towards cyclists who are sometimes difficult to recognize or predict and anomalies that are generally difficult to categorize.

V Conclusion

Superclass	#Instances	mIoU $\uparrow$	mP $\uparrow$	mR $\uparrow$	mF1 $\uparrow$
Pedestrian	16	33.9	44.1	37.3	40.4
Cyclist	22	41.6	58.3	49.5	53.5
Vehicle	736	33.0	48.3	41.0	44.4
Animal	5	0.0	0.0	0.0	0.0
Traffic facility	3,360	28.6	39.9	33.9	36.7
Obstruction	125	20.5	34.0	22.9	27.4
Misc	15	36.7	60.7	37.9	46.7

TABLE II: CODA Superclasses. Evaluation of our approach on seven superclasses. Adapted from [71].

In this work, we have presented complementary learning to detect real-world model failures. We leverage complementary training paradigms to detect contradicting outputs on the same task. This way, we can detect the failure modes of a model in the real-world, far beyond the limited scope of a test split. We demonstrate our approach with a motion segmentation of point clouds for autonomous driving, where we leverage a supervised stream for semantic motion labels and a self-supervised stream for predictive motion labels. By inspecting over 20,000 frames, we showed that our approach exposes model failures in repeating scenarios, which allows for the classification of model failures. In addition, we provide LidarCODA, the first anomaly detection benchmark with labeled point clouds to perform a quantitative analysis. We could show that our approach shows an increased sensitivity of often hard-to-detect and hard-to-predict bicycles as well as hard-to-categorize objects. In the future, we will be interested in the effects of including challenging data points into re-training our models, and potentially fine-tuning our self-supervised stream with the supervised one.

Limitations. When both streams are wrong, model failures go undetected, as shown in Figure 10. This behavior is known and unavoidable [21, 33]. Thus, our approach should not be understood as a stand-alone solution but can be seen as one of many triggers to detect challenging scenarios. In addition, our self-supervised stream depends on a clustering strategy, which can lead to faulty clusters and misclassifications.