CoLA: Conditional Dropout and Language-driven Robust Dual-modal Salient Object Detection

Abstract

The depth/thermal information is beneficial for detecting salient object with conventional RGB images. However, in dual-modal salient object detection (SOD) model, the robustness against noisy inputs and modality missing is crucial but rarely studied. To tackle this problem, we introduce Conditional Dropout and LAnguage-driven(CoLA) framework comprising two core components. 1) Language-driven Quality Assessment (LQA): Leveraging a pretrained vision-language model with a prompt learner, the LQA recalibrates image contributions without requiring additional quality annotations. This approach effectively mitigates the impact of noisy inputs. 2) Conditional Dropout (CD): A learning method to strengthen the model’s adaptability in scenarios with missing modalities, while preserving its performance under complete modalities. The CD serves as a plug-in training scheme that treats modality-missing as conditions, strengthening the overall robustness of various dual-modal SOD models. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art dual-modal SOD models, under both modality-complete and modality-missing conditions. We will release source code upon acceptance.

1 Introduction

Salient object detection (SOD) is a foundational task in computer vision so as to extract the most attractive objects/regions from a scene. Benefiting from the auxiliary inputs like depth and thermal, advanced dual-modal SOD models [46, 43, 9, 48, 61, 6, 57] are able to detect salient objects even in complex and challenging scenes. Due to the ability of redundant elimination and reduction in computational complexity, this technique has been widely applied to enhance the performance of action recognition [24], object tracking [62, 16], semantic segmentation [5, 26], and image fusion [28].

Figure 1: (a): The two examples demonstrate the cases when the RGB or the Depth inputs are noisy, QA means quality assessment. (b): examples of two modality-missing conditions. The proposed CoLA produces robust results.

In general, the high accuracy of a dual-modal SOD model relies on high-quality and complete inputs. However, in real-world situations, 1) the input may be noisy due to communication fault , 2) and one of the modalities may be missing due to device malfunction. A SOD model trained on ideal inputs may degrade when faced with corrupted or partially missing inputs.

As shown in Fig. 1(a), the noisy RGB image or depth/thermal image leads to unsatisfactory predictions in conventional methods such as CSRNet [18] and SPSN [25]. It is necessary to reweight the contribution of each modality rather than treating all inputs as ideal. Though quality-aware SOD models have been proposed, their quality estimation relies on either an off-the-shelf estimation network [9, 48] or pseudo-labels [4, 6]. The parameters of the off-the-shelf estimation network [9] are fixed, and pseudo-labels [4, 6] are inaccurate; neither of them can adapt well to the target dataset such as TAGFNet [48] and DIGR [6]. Recent advances of the pre-trained vision-language model [41] have shown potential capability to assess the quality of an image [49]. To this end, we propose a language-driven quality assessment (LQA) module to assess the quality of each input and reweight the contribution for the network. The LQA contains a fixed prompt and a learnable prompt; only the parameters of the learnable prompt are updated during training, while the fixed prompt and vision/language encoder are frozen. Compared with other image quality assessment methods, this design not only maintains the generalization ability of the pre-trained vision-language model but also adapts to the target dataset in a parameter-efficient fine-tuning manner. As shown in the last column of Fig. 1(a), our predictions with the proposed LQA best fit the GT in both noisy RGB or depth.

Another limitation of existing dual-modal SOD models is that they do not consider performance degradation when the modality is partially missing. This leads to existing models overly relying on complete modal inputs, resulting in poor performance when modality-missing conditions. Furthermore, in system deployment, the phenomenon of Modality-missing occurs naturally. It should be noted that even though the RGB modality is missing from the model’s input, it still exists in principle. As shown in 1(b), when one modality is missing, dual-modal SOD models like LSNet[61] and TAGF[48] can not recognize the salient object in the images, even though another modality image is easy to recognize the salient object. This indicates existing dual-modal SOD models lack consideration for the situation of modality-missing. Resulting in insufficient utilization of the image information of another existing modality when the modality is missing. To address this limitation, we explore a training strategy to enhance the performance of dual-modal SOD under both modality-complete and modality-missing conditions. Inspired by conditional controls [56], we treat the modality missing as the condition and replicate the encoders trained when the modality is complete. Subsequently, we freeze the original encoder and update the parameters of the copied encoder and zero convolutions. This approach, namely Conditional Dropout (CD), preserves the model’s capability when dealing with complete modalities and enables it to extract modality-invariant and specific features.

We propose a new perspective on dual-modal SOD, aiming to investigate a robust framework resistant to noisy and incomplete inputs while maintaining accuracy when facing ideal inputs. In sum, the main contributions and why this work is non-trivial are as follows:

• To the best of our knowledge, this is the first robust dual-modal SOD model against both noisy image and modality-missing.

• A language-driven quality assessment (LQA) module with a learnable prompt is proposed to reweight modality contributions. LQA enhances the robustness and performance of the model with noisy images.

• A Conditional Dropout (CD) learning method is proposed to promote the performance of the model in both modality complete and missing conditions.

• Though the proposed network is simply designed for better scalability, our model outperforms the state-of-the-art in both modality complete and missing conditions.

2 Related Work

2.1 Dual-modal salient object detection

Recent years have seen increased research in dual-modal salient object detection, particularly in RGB-T [45, 43, 18, 44, 19, 50, 9, 48, 61] and RGB-D [6, 57, 8, 25, 39, 33, 30, 31, 32] modalities. These works have made significant progress in the field of salient object detection. For instance, [9, 6] enhanced performance through various estimations, [18, 19] focused on fusion methods like context-guided and early single-stream fusion, and [43, 44] tackled issues like multi-interactive fusion and spatial misalignment. [45, 48, 50, 61] introduced strategies like selective feature collection, compensation for low illumination, cross-scale fusion, and feature propagation. Additionally, [57, 8, 25] contributed with methods like Criss-Cross Dynamic Filter Networks, cross-modality interaction, and Superpixel Prototype Sampling, respectively, to advance salient object detection.

These studies have collectively enhanced dual-modal salient object detection performance. However, these methods frequently show limited adaptability to low-quality or absent modalities. Often, models trained under ideal conditions struggle with generalization in scenarios with noisy or partially missing inputs. To address this, we propose CoLA adept at handling both complete and incomplete modalities. CoLA guarantees robust performance in scenarios with noisy or unavailable modalities.

2.2 Incomplete modality learning

Incomplete modality learning aims to alleviate the modality missing problem due to sensor failures or environmental constraints [12]. Recent advantages can be categorized into reconstruction-based and fusion-based approaches. Reconstruction-based methods exploit the remaining modalities to recover or reconstruct the missing ones. For example, latent space strategies [21] can be used to generate complete multimodal inputs, and Generative Adversarial Network [3, 23, 38] can be employed to restore the absent modalities. Reconstruction-based methods require training and deploying a distinct model for each subset of missing modalities, leading to high complexity in practical applications. Recent works have explored fusion-based methods to learn joint representations directly from incomplete multimodal inputs across different applications. [12, 35, 52, 10] explore fusion methods for incomplete modalities from various perspectives such as regional perception, multi-tasking, and feature space.

These methods tend to make the model adapt to the missing modality, which can hardly ensure unaffected performance on complete modalities. We propose a Conditional Dropout method to improve performance in incomplete modalities without compromising complete modalities, alleviating the missing modality issue in dual-modal SOD tasks.

2.3 Vision-language model and application

Recent works research has extensively investigated a range of vision-language models, e.g., DALL-E [42], ALIGN [20] and CLIP [41], which harmonizes text and image understanding by learning to associate images with their textual descriptions effectively. These models have demonstrated remarkable capabilities across a spectrum of computer vision tasks such as crowd counting [27], pedestrian detection [29], human and object interaction detection [37], scene text detection [55], etc. Beyond recognition, CLIP is studied for monocular depth estimation in a zero-shot manner [58]. Due to the variety of downstream tasks, many fine-tuning improvements based on CLIP have been proposed [60, 59, 54], which have greatly enhanced the versatility of CLIP [51, 55, 54].

Zero-shot and fine-tune based CLIP applications present versatile approaches for diverse computer vision tasks. We utilize CLIP to access the quality of inputs by prompt learning, leveraging its transformable capabilities to tackle the challenges posed by noisy inputs.

Figure 2: The architecture of CoLA represents a two-stage neural network with Stage I training a language quality assessment (LQA) to calibrate feature fusion, and Stage II training with Conditional Dropout enhances the capabilities of both missing and complete modalities.

3 Methodology

3.1 Overview of the framework

The process of training a conventional multi-modal SOD can be summarized as $f(x)\stackrel{{\scriptstyle t_{1}}}{{\leftarrow}}\left\{f;x\right\}$ where $t_{1}$ denotes the training process, $x$ refers to the multi-modal inputs, and $f(x)$ is the trained model with data $x$.

Unlike previous works, our CoLA consists of four components: 1) a dual-branch encoder for feature extraction; 2) an identically structured encoder with a Conditional Dropout ($CD$) for robust modality feature supplementation (in training stage II); 3) a Language-driven Quality Assessment module (LQA) for learning modality contribution ratios (in training stage I); 4) a decoder for feature aggregation to produce the output. Importantly, our encoder and decoder designs are kept simple without complex interactions. This allows us to demonstrate the efficacy of our proposed modules clearly. Based on the components, the training process can be expressed as:

$$f^{\prime}(x)\xleftarrow{t_{1}}\{f^{\prime};LQA(x)\},$$ (1)

$$f(x)\xleftarrow{t_{2}}\{f^{\prime}(\rho(x));CD(\rho(x))\},$$ (2)

where $x$ is the multi-modal inputs, $f^{\prime}(x)$ represents the trained model of the first stage. The first and second stage training processes are denoted by $t_{1}$ and $t_{2}$ respectively, and $\rho(x)$ is the input with missing probability $\rho$.

Figure 3: Architectural comparison of (a) No-Reference Method, (b) Pre-trained Assessment and (c) Our LQA.

3.2 Language-driven quality assessment for low quality image

Fig. 3 compares other image quality assessment methods with LQA. Existing assessment methods can be mainly categorized into two types. The first type (Fig. 3(a)) relies on the inherent characteristics of the image, such as brightness, contrast, and other attributes, to calculate the image quality, such as BRISQUE [36]. The second type uses neural network models pre-trained on other dataset (Fig. 3(b)), for example, GIE[9]. The first type is constrained by its incapacity for training, while the second type can not adapt to the target dataset. Inspired by CLIP-IQA [49], we found vision-language model have demonstrated superior performance in assessing the quality of an image, so we design LQA which based on fine-tuning vision-language model (Fig. 3(c)) for modality quality assessment to reweight the contribution of each modality to extract robust features from dual-modal inputs. The process of LQA is formulated as follows:

$$g=LQA(\mathcal{M},\mathcal{F}(\mathcal{M};\theta),\mathcal{T}),$$ (3)

where $\mathcal{M}=\{m_{1},m_{2}\}\in\mathbb{R}^{3\times H\times W}$ are a pair of dual-modal images, e.g., RGB and Thermal in Fig. 2. $\mathcal{T}$ are texts; $\mathcal{F}$ is the encoder I and II with parameters $\theta$; $g$ are the output of LQA and fed into the decoder. We first pass $m_{1}$ and $m_{2}$ through the image encoder of CLIP to obtain the image embedding $\varepsilon_{i}\in\mathbb{R}^{1\times D}$, where $D$ is the image embedding dimension ($D$ is set to 512). In parallel, the text encoder in CLIP receives inputs $\mathcal{T}$$=\{A\ photo\ of\ high\ quality.\}$ and outputs text embeddings $\varepsilon_{t}\in\mathbb{R}^{1\times D}$. To adapt the target dataset, we add a learnable prompt $\omega$ to the text embedding $\varepsilon_{t}$:

$$\varepsilon_{i}=\varepsilon_{i}+\omega.$$ (4)

In this process, we explore adapting pre-trained vision-language models for quality assessment by adding a small trainable parameter to the text encoder. The parameter $\omega$ is trained to enhance the model’s robustness against noisy images. To this end, we define the quality score estimated by CLIP on the two modality images as $\alpha^{m_{1}},\alpha^{m_{2}}$:

$$\alpha^{m_{1}}=sim(\varepsilon_{i}^{m_{1}},\varepsilon_{t})=\frac{\varepsilon_{i}^{m_{1}}\cdot\varepsilon_{t}}{||\varepsilon_{i}||\ ||\varepsilon_{t}||},\$$ (5)

$$\alpha^{m_{2}}=sim(\varepsilon_{i}^{m_{2}},\varepsilon_{t})=\frac{\varepsilon_{i}^{m_{2}}\cdot\varepsilon_{t}}{||\varepsilon_{i}||\ ||\varepsilon_{t}||},\$$ (6)

where $\varepsilon_{i}^{m_{1}}$ and $\varepsilon_{i}^{m_{2}}$ are the image embeddings from the CLIP image encoder, $sim$ indicates cosine similarity between $\varepsilon_{i}$ and $\varepsilon_{t}$. It should be noted that LQA is trained only in stage I, which is expected to improve the ability to recognize noisy images. Its parameters remain fixed and are not updated in the training stage II. Subsequently, we fuse the individual layers of RGB and T features using the $\alpha$:

$$g_{j}=g_{j}^{m_{1}}*\frac{\alpha^{m_{1}}}{\alpha^{m_{1}}+\alpha^{m_{2}}}+g_{j}^{m_{2}}*\frac{\alpha^{m_{2}}}{\alpha^{m_{1}}+\alpha^{m_{2}}},$$ (7)

where $g_{j}^{m_{1}}$ and $g_{j}^{m_{2}}$ represent the $j$-th layer features of the first and second modality respectively; $g=\{g_{1},...,g_{n}\}$ is the fused feature passed into the decoder; $\alpha^{m_{1}}$ and $\alpha^{m_{2}}$ are quality score for ${m_{1}}$ and ${m_{2}}$. For the decoder, we perform standard CBAM operations on $g_{j}$, then conv and upsample before adding to $g_{j+1}$. Based on our model, $n$ is set to 5.

3.3 Conditional Dropout for incomplete multi-modal learning

Figure 4: Architectural comparison of (a) modality dropout [52] and (b) Conditional Dropout in dual-modal object detection.

Previous methods for handling missing modalities mostly employ direct modality dropout [52] (Fig. 4(a)) during training, which can improve model performance with missing modalities but significantly degrade performance with complete modalities. We proposed Conditional Dropout (Fig. 4(b)) to preserve the capability in modality-complete scenarios by freezing the pre-trained model and finetuning the trainable copy and zero convolution.

In stage II, we will choose one of the input pairs $\rho(\mathcal{M})$ from the following conditions:

$$\rho(\mathcal{M})=\left\{\begin{aligned} Cond_{1}:=\ &\{m_{1},m_{2}\}\\ Cond_{2}:=\ &\{m_{1},\ \phi\ \ \}\\ Cond_{3}:=\ &\{\ \phi\ \ ,m_{2}\}\end{aligned}\right.,$$ (8)

where $\phi$ denotes the zero input, $Cond_{1}$, $Cond_{2}$, and $Cond_{3}$ denote the selection probabilities for three distinct input pairs. The original parameters $\theta$ of the encoder from the stage I are frozen and we duplicate the encoder to create a trainable copy Encoder ${\rm I^{\prime}}$, Encoder ${\rm II^{\prime}}$ with parameters $\theta_{f}$. Then we connect them using zero convolutions $\mathcal{Z}$ with both weight and bias initialized to zero:

$$g=\mathcal{F}(\rho(\mathcal{M});\theta)+\mathcal{Z}(\mathcal{F}(\rho(\mathcal{M});\theta_{f});\theta_{z}),$$ (9)

where $\theta_{z}$ is the parameter of the zero convolution. According to Eq. 9, the first term $\mathcal{F}(\rho(\mathcal{M});\theta)$ represents the frozen model $f$ trained in stage I with parameter $\theta$, which gains the ability to recognize common features from dual modalities. The second term $\mathcal{Z}(\mathcal{F}(\rho(\mathcal{M});\theta_{f});\theta_{z})$ denotes the additional parameters $\theta_{f}$ and $\theta_{z}$ introduced by the Conditional Dropout in the second stage. By using incomplete modalities as input during training, we force the model to extract more fine-grained representations from each modality. In summary, the single modality inputs compel the model to learn richer representations recorded in the second term. Meanwhile, the frozen first term preserves the previously learned knowledge. Finally, the $\mathcal{Z}(\cdot;\cdot)$ with zero initialization ensures minimal influence at the beginning of the training stage II, and the newly learned enriched features are progressively integrated into the model during training. The Conditional Dropout framework paves a way to obtain missing data robustness while maintaining accuracy on complete inputs. This framework could strengthen both single-modality and full-modality potentials concurrently.

3.4 Learning objective

Given a pair of images $m_{1},m_{2}$ and an input prompt $\mathcal{T}$, in the training stage I, we input $m_{1}$ and $\mathcal{T}$ into the proposed LQA module, learning to predict scores $\alpha$ for evaluating image quality as shown in Eq. 5. The $m_{1}$ and $m_{2}$ are also fed into the encoder, and the output $g_{i}^{m_{i}}$ are fused by $\alpha$ in Eq. 7. After passing through the decoder, the loss is calculated between the output and GT. During training, We use BCE loss and IOU loss as our loss functions:

$$\mathcal{L}_{total}=\mathcal{L}_{bce}(pred,GT)+\mathcal{L}_{iou}(pred,GT).$$ (10)

During training stage II, parameters of the initial encoder are frozen, a trainable copy is initialized, and images are fed into both the stage I encoder and the new stage II encoders as per Eq. 9. After feeding into the decoder, the loss is calculated between the output and GT to optimize the parameters of the stage II trainable copy and zero convolution. CoLA focuses on simplicity and extensibility rather than complex model design. The encoder, decoder, and training loss are kept straightforward without meticulous engineering. Combining the above methods, our approach establishes a new evaluation in the dual-modal SOD field, capable of maintaining the model’s accuracy under conditions of noisy and missing inputs.

4 Experiments

4.1 Experiment setup

Dataset. For RGB-T, we employ three widely recognized public datasets for our experiment: VT821 [47], VT1000 [46], and VT5000 [45]. In terms of training, we utilized 2500 image pairs from the VT5000 dataset for training purposes. The model’s performance under modality-complete and modality-missing was evaluated using the remaining 2500 image pairs from the VT5000 dataset, along with the VT821 and VT1000 datasets.
For RGB-D, we utilize four well-known public datasets for our experiment: SIP [15], NJUK [22], DES [7], and NLPR [40]. For training purposes, we used a combination of 1485 image pairs from the NJUK dataset and 700 image pairs from the NLPR dataset.

Implementation details. We use a single NVIDIA GeForce RTX 3090 GPU. The backbone of all methods utilizes a ResNet-50 [17] model pre-trained on the ImageNet [11] dataset. We use the Adam optimizer with an initial learning rate of 1e-4. In stage I, we train the model for a total of 100 epochs with a batch size of 8 and divide the learning rate by 10 every 45 epochs. In stage II, we conduct a training period covering 60 epochs, starting with an initial learning rate set at 1e-4. This learning rate is then decreased by a factor of 10 after every 35 epochs.

Table 1: Quantitative experiments of different RGB-T models in modality-complete and modality-missing conditions. $\uparrow$ denotes that a larger value is better. The best results are in red, the second-best results are in blue, and the third-best results are in green.

Datasets	Conditions		Metric	CSRNet	ADF	TNet	DCNet	LSNet	MIDD	TAGFNet	Ours
	RGB	T		[18]	[45]	[9]	[44]	[61]	[43]	[48]
VT821			$E_{m}$$\uparrow$	0.909	0.841	0.919	0.911	0.910	0.901	0.912	0.922
			$F_{\beta}$$\uparrow$	0.837	0.718	0.823	0.841	0.829	0.819	0.825	0.849
			$E_{m}$$\uparrow$	0.720	0.640	0.834	0.750	0.787	0.799	0.855	0.864
			$F_{\beta}$$\uparrow$	0.540	0.465	0.713	0.644	0.687	0.680	0.727	0.752
			$E_{m}$$\uparrow$	0.726	0.765	0.813	0.872	0.841	0.840	0.861	0.900
			$F_{\beta}$$\uparrow$	0.551	0.655	0.740	0.780	0.749	0.737	0.771	0.817
	Average Drop		$E_{m}$$\uparrow$	-0.186	-0.139	-0.096	-0.100	-0.096	-0.082	-0.049	-0.040
			$F_{\beta}$$\uparrow$	-0.291	-0.158	-0.097	-0.129	-0.111	-0.110	-0.076	-0.065
	Average		$E_{m}$$\uparrow$	0.785	0.749	0.855	0.844	0.846	0.847	0.876	0.895
			$F_{\beta}$$\uparrow$	0.643	0.613	0.759	0.755	0.755	0.745	0.774	0.806
VT1000			$E_{m}$$\uparrow$	0.952	0.928	0.937	0.949	0.951	0.946	0.952	0.955
			$F_{\beta}$$\uparrow$	0.900	0.873	0.885	0.902	0.882	0.884	0.890	0.904
			$E_{m}$$\uparrow$	0.789	0.761	0.896	0.788	0.858	0.886	0.919	0.930
			$F_{\beta}$$\uparrow$	0.634	0.656	0.810	0.723	0.783	0.807	0.834	0.855
			$E_{m}$$\uparrow$	0.781	0.872	0.891	0.943	0.913	0.927	0.936	0.943
			$F_{\beta}$$\uparrow$	0.652	0.801	0.839	0.884	0.853	0.860	0.871	0.887
	Average Drop		$E_{m}$$\uparrow$	-0.167	-0.112	-0.044	-0.083	-0.065	-0.040	-0.025	-0.018
			$F_{\beta}$$\uparrow$	-0.257	-0.145	-0.061	-0.099	-0.064	-0.050	-0.038	-0.027
	Average		$E_{m}$$\uparrow$	0.841	0.854	0.908	0.893	0.907	0.920	0.936	0.943
			$F_{\beta}$$\uparrow$	0.729	0.777	0.845	0.836	0.839	0.851	0.865	0.882
VT5000			$E_{m}$$\uparrow$	0.901	0.887	0.927	0.922	0.914	0.906	0.913	0.927
			$F_{\beta}$$\uparrow$	0.818	0.802	0.827	0.822	0.820	0.807	0.819	0.843
			$E_{m}$$\uparrow$	0.744	0.751	0.854	0.693	0.801	0.819	0.869	0.887
			$F_{\beta}$$\uparrow$	0.523	0.603	0.743	0.572	0.689	0.699	0.742	0.774
			$E_{m}$$\uparrow$	0.733	0.820	0.786	0.895	0.847	0.874	0.895	0.913
			$F_{\beta}$$\uparrow$	0.550	0.707	0.717	0.792	0.752	0.764	0.791	0.822
	Average Drop		$E_{m}$$\uparrow$	-0.163	-0.102	-0.107	-0.128	-0.090	-0.060	-0.031	-0.027
			$F_{\beta}$$\uparrow$	-0.281	-0.147	-0.097	-0.140	-0.100	-0.076	-0.052	-0.045
	Average		$E_{m}$$\uparrow$	0.793	0.819	0.856	0.837	0.854	0.866	0.892	0.909
			$F_{\beta}$$\uparrow$	0.630	0.704	0.762	0.729	0.754	0.757	0.784	0.813

Evaluation metrics. We employ four widely used evaluation metrics to validate the performance of our model and other existing SOTA methods: S-measure($S_{\alpha}$) [13], Mean F-measure ($F_{\beta}$) [1], Mean E-measure($E_{m}$) [14] and Mean Square Error(MAE) [2]. Average Drop and Average, to measure the robustness and overall capability of a dual-modal SOD model respectively. Average Drop refers to the average performance drop when modalities are missing compared to when modalities are complete. Average refers to the average performance across the three conditions.

4.2 Compare with state-of-the-art

To validate the effectiveness of our CoLA in RGB-T SOD under modality-complete and modality-missing, we compared it with nine state-of-the-art methods from the past three years. The compared methods include ADF [45], MIDD [43], CSRNet [18], DCNet [44], TNet [9], TAGFNet [48] and LSNet [61]. We also compared our CoLA with six state-of-the-art methods in RGB-D from the past three years. The compared methods include D3Net [15], DIGR [6], C²DFNet [57], CIRNet [8], SPSN [25], HiDAnet[53].

Our quantitative comparison results are shown in Table 1. The compared models can be classified into two categories: models like CSRNet [18] and ADF [45] exhibit significant performance degradation when either RGB or Thermal modality is missing, indicating poor robustness in handling modality-missing conditions. Some methods like DCNet[44] perform well when one modality is missing but suffer greatly when the other is absent, indicating an excessive reliance on a particular modality.

Our method excels by achieving the best results in both Average Drop and Average across all datasets, demonstrating unparalleled robustness and overall performance. CoLA also achieved the best performance when dealing with modality-complete as well as when dealing with modality-missing.

Table 2: Ablation experiments for each component on the VT5000 dataset. The Baseline is the network with a value of $\alpha$ set to 0.5.

		Modality Complete				Missing RGB				Missing Thermal				Average
		$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$
(a)	Baseline	.859	.892	.801	.052	.820	.866	.727	.064	.845	.883	.793	.058	.841	.880	.774	.058
(b)	Baseline+LQA	.887	.923	.834	.039	.828	.876	.754	.059	.849	.884	.789	.061	.855	.894	.792	.053
(c)	Baseline+CD	.880	.908	.818	.043	.833	.870	.750	.061	.868	.902	.811	.044	.860	.892	.793	.049
(d)	Baseline+LQA+CD	.892	.927	.843	.037	.840	.887	.774	.052	.874	.913	.822	.042	.869	.909	.813	.044

Figure 5: Radar chart of the proposed model and some existing state-of-the-art methods in three datasets. R-M means the performance of the model under RGB-Missing. T-M means the performance of the model under Thermal-Missing.

As shown in Fig. 5, we compared the best five models: MIDD [43], DCNet [44], TNet [9], TAGFNet [48], LSNet [61], and our model to establish a radar chart. From the chart, it can be seen that CoLA achieved state-of-the-art performance across all metrics under the three conditions on all datasets.

Table 3: Compare with other image quality assessment methods for training stage I . The Baseline is the network with a value of $\alpha$ set to 0.5. CLIP-IQA [49] and CLIP-IQA⁺ [49] refer to the use of frozen CLIP and CLIP fine-tuned with the CoOp [60] method, respectively.

		Baseline	+BRISQUE [36]	+GIE    [9]	+CLIP-IQA [49]	+CLIP-IQA+[49]	+LQA
VT821	$S_{\alpha}$$\uparrow$	.844	.854	.870	.869	.878	.888
	$E_{m}$$\uparrow$	.873	.893	.910	.898	.910	.915
	$F_{\beta}$$\uparrow$	.805	.797	.812	.796	.830	.839
	MAE$\downarrow$	.055	.046	.044	.050	.042	.038
VT1000	$S_{\alpha}$$\uparrow$	.897	.909	.920	.913	.918	.924
	$E_{m}$$\uparrow$	.924	.939	.944	.945	.949	.955
	$F_{\beta}$$\uparrow$	.854	.867	.880	.876	.874	.904
	MAE$\downarrow$	.038	.034	.030	.031	.030	.024
VT5000	$S_{\alpha}$$\uparrow$	.859	.864	.866	.878	.882	.887
	$E_{m}$$\uparrow$	.892	.901	.914	.916	.915	.923
	$F_{\beta}$$\uparrow$	.801	.810	.819	.822	.825	.834
	MAE$\downarrow$	.052	.048	.047	.042	.040	.039

4.3 Ablation studies

Effectiveness of each component in our model. Table 2 summarizes how progressively incorporating our proposed modules improves model performance under modality-complete and modality-missing. Comparing (a) and (b) shows that adding the LQA module helps CoLA better fuse information from the two modalities to handle noisy images, thus achieving better results under modality-complete condition. Contrasting (c) and (d), the model trained without LQA in stage I will be weaker than the model trained with LQA, which means employing only CD in stage II results in weaker performance compared to (d). By incorporating both LQA and Conditional Dropout, the model significantly enhances its ability to extract valuable information from each individual modality. This fundamentally strengthens modal robustness, enabling CoLA to minimize performance loss when modalities are missing by better utilizing the available single modalities.

The efficiency of LQA. Table 3 compares other image quality assessment methods with LQA. In addition to the No-Reference[36] and Pre-trained quality assessment networks[9]. We also compared the CLIP-IQA [49], which is based on CLIP[41], to assess image quality. The results indicate that compared to using a fixed threshold of 0.5, employing image quality assessment methods such as BRISQUE[36], GIE[9], CLIP-IQA [49], etc., can lead to some performance improvement. However, due to the lack of inherent learning capability and generalization to datasets, these methods have limited impact.

For better demonstrate the effectiveness of LQA, we extract all noisy images like 1(a) line 1 from original VT821 to create a new dataset VT821-noisy with 76 images. Experiments were conducted on this dataset using all compared methods, with results provided in the supplementary materials.

Table 4: Ablation experiments of the Conditional Dropout module on the VT5000 dataset, “Copy" denotes duplicating encoder, “Freeze" denotes freezing modules except the copied encoder, and “MD" denotes Modality Dropout. “Z-Conv” denotes Zero-Convolution.

	Copy	Z-Conv	Freeze	MD	Modality Complete				Missing RGB				Missing Thermal				Average
					$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$
(a)					.887	.923	.834	.039	.828	.876	.754	.059	.849	.884	.789	.061	.855	.894	.792	.053
(b)	$\checkmark$				.872	.900	.813	.049	.815	.860	.735	.066	.851	.877	.788	.062	.857	.894	.786	.054
(c)	$\checkmark$	$\checkmark$			.878	.910	.819	.046	.820	.863	.733	.062	.847	.875	.785	.066	.857	.894	.786	.054
(d)				$\checkmark$	.867	.902	.805	.051	.839	.880	.759	.057	.865	.899	.795	.054	.857	.894	.786	.054
(e)	$\checkmark$	$\checkmark$		$\checkmark$	.872	.913	.822	.046	.834	.875	.749	.061	.868	.901	.803	.049	.858	.896	.791	.052
(f)	$\checkmark$	$\checkmark$	$\checkmark$		.890	.923	.836	.038	.822	.869	.743	.064	.855	.889	.795	.056	.856	.894	.791	.053
(g)	$\checkmark$		$\checkmark$	$\checkmark$	.889	.923	.832	.039	.837	.883	.765	.054	.868	.905	.805	.046	.865	.904	.801	.046
(h)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	.892	.927	.843	.037	.840	.887	.774	.052	.874	.913	.822	.042	.869	.909	.813	.044

Ablation study for CD. Ablation experiments were conducted in this module to validate the effectiveness of the proposed Conditional Dropout. As shown in Table 4, observing (a) indicates that the original model lacks modality robustness, as evidenced by the significant performance reduction under modality-missing. Comparing (a) and (d) shows that simply applying Modality Dropout can improve model performance under modality-missing, but this leads to worse performance under modality-complete. Contrasting (d) and (e) demonstrates that adding the Copy operation can effectively reduce the performance decline caused by Modality Dropout under modality-complete. A comparison between (e) and (h) reveals that the absence of the freeze operation impacts the encoder’s ability to effectively extract features, leading to diminished performance under modality-complete. Analyzing (f) and (h) suggests that employing Copy and Freeze without Modality Dropout does not enhance model robustness.

Figure 6: Comparison of Models Trained with Conditional Dropout, Modality Dropout, and Original Model in VT5000 dataset.

Conditional Dropout as a plug-in. We conducted generalization experiments to validate the effectiveness of Conditional Dropout. The results in Fig. 6 show that LSNet [61] and T-Net [9] augmented with Conditional Dropout achieves significantly improved robustness under modality-missing while the performance under modality-complete remains almost unaffected. These experiments validate the generalization of Conditional Dropout in advancing the robustness of dual-modal methods.

Extended experiments under modality-complete with CD. As shown in Table 5, the model demonstrates improved performance under modality-complete after applying Conditional Dropout, especially on the VT821 dataset. Comparing (a) with (b) and (d) reveals that simply extending training epochs or copying parameters fails to enhance performance under modality-complete. Contrasting (c) to (a) indicates that directly utilizing Modality Dropout damages performance under complete modalities. Comparing (c) and (e), it can be observed that Conditional Dropout, as opposed to Modality Dropout, can significantly enhance the model’s performance in modality-complete.

Overall, simply increasing model parameters (copying encoders), training epochs, or using Modality Dropout does not improve the model’s performance under modality-complete. Conditional Dropout enhances the model’s ability to extract information from individual modality, allowing the model to maintain or even improve its performance under modality-complete.

Table 5: Extended experiments under modality-complete. “AT" denotes additional training for 60 epochs.

	AT	Copy	Z-Conv	Freeze	MD	VT821				VT1000				VT5000
						$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$
(a)						.888	.915	.839	.038	.924	.955	.893	.024	.887	.923	.834	.039
(b)	$\checkmark$					.870	.881	.806	.051	.915	.944	.879	.031	.871	.910	.810	.046
(c)	$\checkmark$				$\checkmark$	.851	.872	.788	.055	.910	.935	.870	.034	.867	.902	.805	.051
(d)	$\checkmark$	$\checkmark$	$\checkmark$			.878	.905	.832	.043	.922	.948	.879	.029	.878	.910	.819	.042
(e)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	.897	.922	.849	.031	.928	.955	.904	.024	.892	.927	.843	.037

5 Conclusion

In this paper, we proposed a robust dual-modal SOD method to enhance model performance in noisy and modality-missing environments. We adeptly adjust reweighted image contributions by the proposed LQA, which is crucial for noise handling. This method bolsters noise robustness across diverse environments. We further introduce a Conditional Dropout training scheme, effective in processing missing modalities, thus improving model performance with incomplete and complete inputs. Our approach sets a new evaluation in dual-modal SOD, prompting discussions on handling robustness in the SOD field.

CoLA: Conditional Dropout and Language-driven Robust Dual-modal Salient Object Detection
<Supplementary Material>

Shuang Hao^$\star$0009-0007-4621-8485 Chunlin Zhong^$\star$0009-0006-7070-3504 He Tang^🖂0000-0002-8454-1407

Our supplementary material is divided into three parts, Appendix 0.A, Appendix 0.B and Appendix 0.C, focusing on RGB-T inputs, RGB-D inputs and limitation respectively, with the following organization.

Experiments on changing the backbone are in Sec. 0.A.5; in Sec. 0.A.1, We have provided some examples to demonstrate the reliability of the $\alpha$ generated by LQA. In Sec. 0.A.2, we appended Conditional Dropout to VT1000 and VT821 datasets to further prove the generalizability of our method. For fairness in input methods, in Sec. 0.A.3, we retrained all comparison methods using modality dropout and compared them with our method CoLA; in Sec. 0.A.4, we introduced the calculation methods and meanings of Average and Average Dropout mentioned in Table 1. Sec. 0.A.6 describes related experiments for dividing the VT821-noisy subset. Sec. 0.B.1 is a comparison of CoLA with the SOTA methods in the RGB-D field, and Sec. 0.B.2 involves a qualitative comparison with the SOTA methods. Appendix 0.C discusses the limitations of our method and presents examples where it did not perform well.

Appendix 0.A More Experiments of RGB-T inputs

0.A.1 Quantitative Evaluation of LQA

As shown in the Fig 7, we present the quality assessment of LQA for different images. In the images of the first and last rows, the quality of the RGB or Thermal images is extremely low, rendering them unable to provide effective information. Correspondingly, the $\beta_{rgb}$ also approach 0 or 1, indicating that LQA can effectively handle these two extreme quality image scenarios. In the second to fourth lines, it can be observed that the quality of RGB images improves in comparison to thermal images, and the corresponding alpha values also increase. In conclusion, it has been demonstrated that LQA is not only effective in assessing the contribution of noisy images but also performs well on general images.

Figure 7: Different cases of our LQA, $\beta_{rgb}=\frac{\alpha^{rgb}}{\alpha^{rgb}+\alpha^{t}}$,$\beta_{t}=\frac{\alpha^{t}}{\alpha^{rgb}+\alpha^{t}}$.

0.A.2 More Implementations of Conditional Dropout as a Plug-in

We also applied Conditional Dropout to VT1000 and VT821 to validate its effectiveness, and the experimental results are shown in Fig. 8. We found that with Conditional Dropout, model’s performance remains almost unchanged when all modalitiy-complete. Furthermore, its performance in two modality-missing conditions is significantly better than the original model and model using Modality Dropout. This effect is consistent across three different datasets, further demonstrating that Conditional Dropout enhances the model’s robustness and reduces performance degradation in modality-missing conditions.

0.A.3 Comparative Analysis Under Modality Dropout

Using the modality dropout training method may bring certain improvements when modalities are missing, but it can harm the performance under complete modality. Our proposed Conditional Dropout can simultaneously enhance the model’s performance in both missing and complete modality scenarios. To further validate the effectiveness of our CoLA in RGB-T Salient Object Detection (SOD) under both modality-complete and modality-missing scenarios, we compared it with all the methods mentioned in the Sec. 4 under modality dropout training. The compared methods include ADF [45], MIDD [43], CSRNet [18], DCNet [44], TNet [9], TAGFNet [48] and LSNet [61].

The quantitative comparison results are shown in Table 8. Compared with Table 1, comparative methods are mainly divided into two categories. One is that after using Modality Dropout, it can enhance its performance when modality is missing, but it will impair the model’s performance when the modality is complete such as DCNet [44] and LSNet [61]. The other methods like TAGFNet [48] is that using Modality Dropout will impair performance in both situations. Our method, even when Modality Dropout is used in other methods, has achieved the best results in both cases of modality-missing.

Figure 8: Comparison of Models Trained with Conditional Dropout, Modality Dropout, and Original Model in VT1000 and VT821 datasets.

Table 6: Comparison Results of Different Methods on the VT821-noisy Dataset.

Methods	VT821-noisy
	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$
CSRNet [18]	0.736	0.765	0.632	0.110
ADF [45]	0.579	0.564	0.356	0.147
TNet [9]	0.807	0.826	0.729	0.055
DCNet [44]	0.818	0.853	0.752	0.056
LSNet [61]	0.804	0.823	0.732	0.052
MIDD [43]	0.812	0.814	0.705	0.073
TAGFNet [48]	0.772	0.794	0.651	0.074
Ours	0.850	0.888	0.787	0.045

Table 7: Performance of the proposed model under various backbones.

Condition		Backbone	VT821				VT1000				VT5000
RGB	T		$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$
		ResNet50	0.897	0.923	0.852	0.031	0.928	0.956	0.896	0.024	0.892	0.927	0.843	0.037
		ResNet101	0.904	0.927	0.858	0.030	0.929	0.953	0.890	0.024	0.893	0.929	0.844	0.036
		Swin-T	0.899	0.920	0.850	0.030	0.927	0.955	0.895	0.026	0.894	0.930	0.853	0.036
		Swin-S	0.899	0.922	0.854	0.028	0.929	0.958	0.893	0.023	0.895	0.934	0.850	0.034
		Swin-B	0.905	0.930	0.860	0.026	0.935	0.962	0.901	0.020	0.906	0.941	0.857	0.028
		Resnet50	0.809	0.865	0.757	0.053	0.886	0.930	0.858	0.041	0.831	0.885	0.774	0.052
		Resnet101	0.811	0.860	0.749	0.051	0.889	0.929	0.861	0.042	0.829	0.882	0.765	0.053
		Swin-T	0.808	0.862	0.750	0.055	0.885	0.919	0.847	0.047	0.825	0.872	0.770	0.055
		Swin-S	0.820	0.867	0.755	0.056	0.890	0.928	0.855	0.039	0.840	0.885	0.769	0.054
		Swin-B	0.824	0.874	0.766	0.049	0.895	0.935	0.863	0.040	0.840	0.894	0.782	0.047
		Resnet50	0.872	0.900	0.816	0.043	0.914	0.943	0.882	0.030	0.874	0.914	0.822	0.043
		Resnet101	0.877	0.895	0.811	0.042	0.916	0.940	0.884	0.031	0.872	0.910	0.817	0.045
		Swin-T	0.869	0.889	0.805	0.045	0.908	0.933	0.873	0.035	0.865	0.905	0.820	0.047
		Swin-S	0.880	0.900	0.815	0.039	0.920	0.939	0.880	0.028	0.880	0.915	0.830	0.042
		Swin-B	0.883	0.906	0.828	0.037	0.926	0.952	0.890	0.025	0.885	0.922	0.832	0.038

0.A.4 Evaluation Metrics of Average and Average dropout

In this section, we introduce two metrics mentioned in the main text, Average and Average Dropout. Their calculations are simple, but they intuitively reflect the overall performance of the model in situations of modal absence and completeness, demonstrating the strength or weakness of the model’s robustness.

Average. For each metric $F$, we calculate its scores $F_{full}$, $F_{m1}$, $F_{m2}$ under modality-complete, missing $m_{2}$ and missing $m_{1}$. Then we simply compute their average:

$$\mathcal{A}=\frac{F_{full}+F_{m1}+F_{m2}}{3},$$ (11)

$\mathcal{A}$ is the calculated metric Average, which represents the comprehensive performance of the model under both missing and complete conditions.

Average dropout. For Average Dropout, we aim to reflect the performance degradation of the model under missing conditions relative to the full modality performance. Thus, we adopt the following calculation method:

$$\mathcal{D}=\frac{(F_{m1}-F_{full})+(F_{m2}-F_{full})}{2},$$ (12)

where $\mathcal{D}$ denotes Average Dropout. As can be seen, if the model suffers significant performance loss under the missing modality, this can be reflected through Average Dropout.

0.A.5 Experiments on Various Backbones

In our study, we have conducted a series of experiments to evaluate the performance of prevalent backbone architectures in the context of saliency detection, which includes ResNet50 [17], ResNet101 [17], and Swin Transformer [34] variants (Swin-T, Swin-S, Swin-B). We performed these evaluations under conditions of both complete and missing modalities as shown in Table 7, encompassing all three datasets within the RGB-T scope. It can be observed that compared to ResNet50, ResNet101 has shown some improvements in results. Within the experiments involving Swin architectures, Swin-B achieved the best outcomes, although Swin-T and Swin-S did not exhibit significant changes compared to ResNet50. The systematic experimentation across different backbones and modalities aims to discern the robustness and adaptability of our framework.

0.A.6 Experiments on Noisy Inputs

VT821-Noisy. To better assess the effectiveness of our proposed LQA module, we aggregate pairs of images with inherent noise from all RGB images in the VT821 dataset (referred to as VT821-Noisy), which is a sub-set of the original VT821.

Table 8: Quantitative experiments of different RGB-T models trained by Modality Dropout in modality-complete and modality-missing conditions. $\uparrow$ denotes that a larger value is better. The best results are in red, the second-best results are in blue, and the third-best results are in green.

Datasets	Conditions		Metric	CSRNet	ADF	TNet	DCNet	LSNet	MIDD	TAGFNet	Ours
	RGB	T		[18]	[45]	[9]	[44]	[61]	[43]	[48]
VT821			$E_{m}$$\uparrow$	0.840	0.847	0.900	0.911	0.911	0.903	0.886	0.922
			$F_{\beta}$$\uparrow$	0.717	0.752	0.820	0.834	0.815	0.822	0.803	0.849
			$E_{m}$$\uparrow$	0.727	0.785	0.815	0.846	0.859	0.804	0.782	0.864
			$F_{\beta}$$\uparrow$	0.558	0.653	0.720	0.731	0.725	0.683	0.636	0.752
			$E_{m}$$\uparrow$	0.763	0.809	0.847	0.885	0.863	0.855	0.802	0.900
			$F_{\beta}$$\uparrow$	0.595	0.701	0.761	0.788	0.760	0.758	0.671	0.817
	Average Drop		$E_{m}$$\uparrow$	-0.095	-0.050	-0.059	-0.046	-0.040	-0.074	-0.094	-0.040
			$F_{\beta}$$\uparrow$	-0.140	-0.075	-0.069	-0.074	-0.073	-0.102	-0.150	-0.065
	Average		$E_{m}$$\uparrow$	0.776	0.814	0.854	0.881	0.874	0.854	0.823	0.895
			$F_{\beta}$$\uparrow$	0.623	0.702	0.775	0.784	0.770	0.754	0.703	0.806
VT1000			$E_{m}$$\uparrow$	0.902	0.923	0.925	0.937	0.939	0.937	0.929	0.955
			$F_{\beta}$$\uparrow$	0.815	0.853	0.861	0.881	0.870	0.876	0.859	0.895
			$E_{m}$$\uparrow$	0.806	0.865	0.884	0.918	0.916	0.892	0.818	0.930
			$F_{\beta}$$\uparrow$	0.667	0.773	0.810	0.837	0.809	0.812	0.674	0.855
			$E_{m}$$\uparrow$	0.810	0.902	0.910	0.907	0.915	0.904	0.824	0.943
			$F_{\beta}$$\uparrow$	0.691	0.830	0.845	0.867	0.850	0.862	0.692	0.881
	Average Drop		$E_{m}$$\uparrow$	-0.094	-0.040	-0.028	-0.025	-0.024	-0.039	-0.108	-0.018
			$F_{\beta}$$\uparrow$	–0.136	-0.052	-0.034	-0.029	-0.041	-0.039	-0.176	-0.027
	Average		$E_{m}$$\uparrow$	0.839	0.897	0.906	0.921	0.923	0.911	0.857	0.943
			$F_{\beta}$$\uparrow$	0.724	0.818	0.836	0.862	0.843	0.850	0.742	0.877
VT5000			$E_{m}$$\uparrow$	0.828	0.874	0.910	0.903	0.902	0.897	0.891	0.927
			$F_{\beta}$$\uparrow$	0.677	0.764	0.815	0.812	0.795	0.798	0.796	0.843
			$E_{m}$$\uparrow$	0.743	0.802	0.822	0.854	0.857	0.823	0.784	0.887
			$F_{\beta}$$\uparrow$	0.536	0.660	0.746	0.748	0.728	0.701	0.614	0.774
			$E_{m}$$\uparrow$	0.750	0.853	0.865	0.880	0.885	0.880	0.802	0.913
			$F_{\beta}$$\uparrow$	0.555	0.736	0.756	0.781	0.768	0.773	0.652	0.822
	Average Drop		$E_{m}$$\uparrow$	-0.082	-0.047	-0.067	-0.036	-0.031	-0.046	-0.098	-0.027
			$F_{\beta}$$\uparrow$	-0.131	-0.066	-0.064	-0.048	-0.047	-0.061	-0.163	-0.045
	Average		$E_{m}$$\uparrow$	0.773	0.843	0.866	0.879	0.881	0.867	0.826	0.909
			$F_{\beta}$$\uparrow$	0.589	0.720	0.772	0.780	0.764	0.757	0.687	0.813

Figure 9: Qualitative comparisons of the proposed model and existing state-of-the-art methods in three situations. The red or green crosses on the image indicate that RGB or Depth modality is missing.

Evaluation. We tested various RGB-T methods and our method for their performance on the full modality of VT821-noisy. As shown in Table 6, compared to the test results with the complete VT821 dataset (Table 1), our model demonstrated greater advantages, indicating that our model has more significant effects when dealing with noisy images.

Appendix 0.B More Experiments of RGB-D inputs

0.B.1 Quantitative Evaluation

Our RGB-D quantitative comparison results are shown in Table 9 and 10. The compared methods include D3Net [15], DIGR [6], C²DFNet [57], CIRNet [8], SPSN [25], HiDAnet[53]. Compared to RGB-T, the performance degradation of RGB-D methods in the missing of depth is significantly smaller than when RGB is missing. This indicates that in the RGB-D domain, RGB holds a dominant position over depth, and therefore, comparative methods experience significant performance loss when the RGB modality is missing. Our model achieves near-optimal performance in both missing modalities. With evaluation metrics that reflect the model’s robustness and overall capability reaching the highest levels in Average and Average Drop metrics. This demonstrates that our model has the best robustness and comprehensive capability.

Table 9: Validation experiment results of different RGB-D models in modality-complete and modality-missing conditions with SIP,NLPR and DES datasets.

Condition		Method	SIP				NLPR				DES
RGB	Depth		$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$
		D3Net	0.860	0.895	0.844	0.061	0.912	0.944	0.887	0.030	0.898	0.938	0.880	0.031
		DIGR	0.882	0.927	0.881	0.051	0.931	0.950	0.896	0.020	0.936	0.962	0.922	0.019
		C2DFNet	0.872	0.910	0.866	0.054	0.927	0.962	0.907	0.022	0.919	0.940	0.905	0.020
		CIRNet:	0.888	0.907	0.855	0.053	0.933	0.949	0.897	0.023	0.932	0.943	0.905	0.025
		SPSN	0.890	0.914	0.896	0.042	0.926	0.946	0.914	0.022	0.938	0.950	0.943	0.016
		HIDANet:	0.899	0.932	0.893	0.040	0.930	0.959	0.909	0.022	0.947	0.979	0.941	0.014
		Ours:	0.895	0.935	0.894	0.042	0.935	0.957	0.909	0.021	0.935	0.963	0.925	0.018
		D3Net	0.720	0.699	0.689	0.112	0.742	0.730	0.695	0.099	0.758	0.774	0.725	0.105
		DIGR	0.699	0.635	0.534	0.146	0.661	0.603	0.422	0.119	0.668	0.586	0.411	0.122
		C2DFNet	0.687	0.606	0.538	0.124	0.621	0.560	0.538	0.114	0.604	0.568	0.509	0.106
		CIRNet	0.787	0.750	0.721	0.115	0.847	0.823	0.753	0.061	0.893	0.867	0.814	0.047
		SPSN	0.740	0.758	0.725	0.106	0.709	0.700	0.599	0.070	0.768	0.793	0.731	0.051
		HIDANet	0.813	0.854	0.805	0.077	0.823	0.878	0.779	0.063	0.857	0.907	0.832	0.050
		Ours	0.843	0.863	0.822	0.076	0.865	0.903	0.809	0.053	0.909	0.947	0.874	0.030
		D3Net	0.831	0.871	0.795	0.079	0.903	0.923	0.855	0.033	0.870	0.874	0.811	0.040
		DIGR	0.797	0.824	0.779	0.086	0.887	0.902	0.847	0.036	0.850	0.837	0.790	0.042
		C2DFNet	0.812	0.858	0.800	0.077	0.911	0.940	0.882	0.031	0.882	0.902	0.856	0.029
		CIRNet	0.847	0.869	0.825	0.076	0.911	0.925	0.876	0.035	0.873	0.860	0.825	0.042
		SPSN	0.844	0.894	0.837	0.065	0.902	0.934	0.868	0.031	0.884	0.908	0.854	0.033
		HIDANet	0.856	0.900	0.848	0.063	0.907	0.938	0.877	0.032	0.895	0.922	0.875	0.031
		Ours	0.863	0.908	0.855	0.057	0.916	0.946	0.885	0.029	0.900	0.926	0.880	0.030
Average		D3Net	0.804	0.822	0.776	0.084	0.852	0.866	0.812	0.054	0.842	0.862	0.805	0.059
		DIGR	0.793	0.795	0.731	0.094	0.826	0.818	0.722	0.059	0.818	0.795	0.708	0.061
		C2DFNet	0.790	0.791	0.735	0.085	0.820	0.821	0.776	0.056	0.802	0.803	0.757	0.052
		CIRNet	0.841	0.842	0.800	0.081	0.897	0.899	0.842	0.040	0.899	0.890	0.848	0.038
		SPSN	0.825	0.855	0.819	0.071	0.846	0.860	0.794	0.041	0.863	0.884	0.843	0.033
		HIDANet	0.862	0.895	0.849	0.060	0.887	0.925	0.855	0.039	0.900	0.936	0.883	0.032
		Ours	0.867	0.902	0.857	0.058	0.905	0.935	0.868	0.034	0.915	0.945	0.893	0.026
Average Drop		D3Net	-0.085	-0.110	-0.102	0.035	-0.090	-0.118	-0.112	0.036	-0.084	-0.114	-0.112	0.042
		DIGR	-0.134	-0.198	-0.225	0.065	-0.157	-0.198	-0.262	0.057	-0.177	-0.251	-0.322	0.063
		C2DFNet	-0.123	-0.178	-0.197	0.047	-0.161	-0.212	-0.197	0.051	-0.176	-0.205	-0.223	0.048
		CIRNet	-0.071	-0.098	-0.082	0.043	-0.054	-0.075	-0.083	0.025	-0.049	-0.080	-0.086	0.020
		SPSN	-0.098	-0.088	-0.115	0.044	-0.121	-0.129	-0.181	0.029	-0.112	-0.099	-0.151	0.026
		HIDANet	-0.055	-0.055	-0.067	0.030	-0.065	-0.051	-0.081	0.026	-0.071	-0.065	-0.088	0.027
		Ours	-0.042	-0.050	-0.056	0.025	-0.045	-0.033	-0.062	0.020	-0.031	-0.027	-0.048	0.012

Table 10: Validation experiment results of different RGB-D models in modality-complete and modality-missing conditions with NJU2K and STERE datasets.

Condition		Method	NJU2K				STERE
RGB	Depth		$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$	$S_{\alpha}$$\uparrow$	$E_{m}$$\uparrow$	$F_{\beta}$$\uparrow$	MAE$\downarrow$
		D3Net	0.900	0.921	0.890	0.041	0.899	0.925	0.879	0.041
		DIGR	0.933	0.941	0.900	0.026	0.914	0.934	0.880	0.035
		C2DFNet	0.907	0.936	0.898	0.039	0.903	0.937	0.884	0.038
		CIRNet	0.925	0.933	0.895	0.035	0.914	0.930	0.874	0.038
		SPSN	0.912	0.936	0.912	0.033	0.906	0.936	0.898	0.035
		HIDANet	0.928	0.956	0.924	0.028	0.910	0.945	0.891	0.034
		Ours	0.934	0.947	0.913	0.029	0.908	0.941	0.889	0.039
		D3Net	0.768	0.688	0.640	0.105	0.619	0.599	0.525	0.158
		DIGR	0.646	0.598	0.486	0.170	0.585	0.537	0.374	0.190
		C2DFNet	0.673	0.704	0.636	0.151	0.667	0.649	0.620	0.145
		CIRNet	0.803	0.768	0.733	0.109	0.642	0.617	0.514	0.158
		SPSN	0.700	0.703	0.651	0.117	0.615	0.634	0.529	0.139
		HIDANet	0.802	0.826	0.769	0.082	0.663	0.732	0.611	0.134
		Ours	0.856	0.878	0.814	0.066	0.725	0.773	0.659	0.120
		D3Net	0.883	0.904	0.847	0.054	0.892	0.915	0.855	0.051
		DIGR	0.850	0.861	0.821	0.063	0.876	0.894	0.855	0.052
		C2DFNet	0.878	0.911	0.868	0.048	0.890	0.922	0.876	0.041
		CIRNet	0.892	0.900	0.872	0.056	0.891	0.901	0.869	0.055
		SPSN	0.893	0.915	0.870	0.043	0.896	0.929	0.879	0.041
		HIDANet	0.894	0.929	0.886	0.048	0.897	0.931	0.876	0.042
		Ours	0.894	0.923	0.874	0.050	0.900	0.933	0.874	0.040
Average		D3Net	0.850	0.838	0.792	0.067	0.803	0.813	0.753	0.083
		DIGR	0.810	0.800	0.736	0.086	0.792	0.788	0.703	0.092
		C2DFNet	0.819	0.850	0.801	0.079	0.820	0.836	0.793	0.075
		CIRNet	0.873	0.867	0.833	0.067	0.816	0.816	0.752	0.084
		SPSN	0.835	0.851	0.811	0.064	0.806	0.833	0.769	0.072
		HIDANet	0.875	0.904	0.860	0.053	0.823	0.869	0.793	0.070
		Ours	0.893	0.916	0.867	0.048	0.844	0.882	0.804	0.066
Average Drop		D3Net	-0.075	-0.125	-0.147	0.039	-0.144	-0.168	-0.189	0.064
		DIGR	-0.185	-0.212	-0.247	0.091	-0.184	-0.219	-0.266	0.086
		C2DFNet	-0.132	-0.129	-0.146	0.061	-0.125	-0.152	-0.136	0.055
		CIRNet	-0.078	-0.099	-0.093	0.048	-0.148	-0.171	-0.183	0.069
		SPSN	-0.116	-0.127	-0.152	0.047	-0.151	-0.155	-0.194	0.055
		HIDANet	-0.080	-0.078	-0.097	0.037	-0.114	-0.114	-0.148	0.054
		Ours	-0.054	-0.047	-0.069	0.029	-0.096	-0.088	-0.113	0.041

0.B.2 Qualitative Evaluation

Fig. 9 provides a visualization of RGB-D. When all modalities are available, our model demonstrates good performance, even in images with poor depth quality. In the absence of depth or RGB, our model outperforms other models in extracting valuable information from the remaining modalities. Particularly, when RGB is missing, our model can still extract significant objects from the depth image, which other comparative methods cannot achieve.

Appendix 0.C Limitations and Future Works

Our model has effectively addressed issues of low-quality image inputs and modality-missing in the dual-modal salient object detection domain. In practice, these problems persist in other communities such as multi-modal instance segmentation and semantic segmentation. In the future, we will continue to explore the issue of modality-missing in other multi-modal domains.