Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization

To mitigate confirmation bias, it is crucial to filter out noisy samples that are more likely to be false positives. As depicted in Figure 2, two predicted maps of the same sample are generated by dual teachers. It is clear that samples with higher reliability can be identified when the two predicted maps exhibit higher similarity, i.e., there is more agreement and consensus between dual teachers, then the sample is reserved for pseudo-labelling. Conversely, when there is a significant discrepancy between the two maps, the sample will be considered as a false positive, such as frames without distinguishable sound objects or sounds that cannot be accurately represented by a bounding box (e.g., wind sounds), such samples are rejected and discarded.

By intersecting the foreground regions of two predicted maps on the filtered samples, one can generate positional pseudo-labels, named Intersection of Pseudo-Labels (IPL), to guide students’ learning. With the pre-defined foreground threshold $\delta$, two predicted maps $\mathcal{P}_{i}^{t,A},\mathcal{P}_{i}^{t,B}$ could be transferred to binary maps $\mathcal{M}_{i}^{t,A}$ and $\mathcal{M}_{i}^{t,B}$. Weak augmentation $\alpha(\cdot)$ is employed for teachers to generate high-quality pseudo-labels:

We adopt the Intersection over Union (IoU) metric to quantify the similarity between the two maps $\mathcal{M}_{i}^{t,A}$, $\mathcal{M}_{i}^{t,B}$. If the IoU score exceeds the threshold $\tau$, the sample will be accepted, and the intersection of those two maps will be generated as its pseudo-label (IPL). Otherwise, it will be filtered out as a noise sample.

The newly selected unlabeled dataset is applied to the student model along with the corresponding high-quality IPL.

To suppress confirmation bias more effectively, we mix labeled and new unlabeled datasets. Both ground-truth annotations and high-quality IPL are employed to train the student models:

In addition, we incorporate consistency regularization Laine and Aila (2016) in the semi-supervised learning process. Specifically, for a given sample, we obtain IPL from the teachers on weakly augmented images while strong augmentations are applied for samples of students. By enforcing consistency between IPL and students’ predictions, DMT could be more stable with better generalization ability.

Similar to the AVSL method mentioned in Section 3, students are also trained by audio-visual correspondence of contrastive learning loss. Here, we introduce an attention module to add attention to the sounding region in the frame:

Then, the full semi-supervised loss could be derived with $\mathcal{L}_{\text{sup}}$ (see Eq. (13)) on $\mathcal{D}_{mix}$ and $\mathcal{L}_{\text{unsup}}$ (see Eq. (D)) on $\mathcal{D}_{u}$:

Students are updated via gradient descent of $\mathcal{L}_{\text{full}}$, while dual teachers are updated through the exponential moving average (EMA) of corresponding students:

The slowly progressing teachers can be regarded as the ensemble of students in recent training iterations, which enables stable progress in training.

We conduct experiments on two large-scale audio-visual datasets: Flickr-SoundNet Senocak et al. (2018, 2019) and VGG Sound Source Chen et al. (2020d), where there are 5,000 and 5,158 bounding-box annotated samples, respectively. For labeled data, we randomly select 4,250 for training, 500 for validating, and keep the same test sets with 250 samples as previous works Mo and Morgado (2022a, b); Chen et al. (2021b); Liu et al. (2022a). Moreover, we select a subset of 10k and 144k unlabeled samples to train as before. Details can be found in Appendix B.1.

For visual backbones, we followed prior work and used ResNet-18 He et al. (2016) pre-trained on ImageNet Deng et al. (2009). For audio backbones, we select the pre-trained VGGish Hershey et al. (2017) and SoundNet Aytar et al. (2016) with semantic audio information. Further details can be found in Appendix B.2.

We report the Consensus Intersection over Union (CIoU) and Area Under Curve (AUC), following previous settings Senocak et al. (2018, 2019). CIoU represents the localization accuracy, samples with IoU above the threshold $\delta=0.5$ are considered to be accurately located. Considering small objects, we introduce Mean Square Error (MSE), which measures the average pixel-wise difference between two maps without binarization, making it more suitable for evaluating dense prediction tasks on small objects. More details are shown in Appendix B.3.

For audio clips, we pass 96 × 64 log-mel spectrograms to VGGish, and the output is a 512D vector, while the raw waveform of the original 3s audio clip is sent to SoundNet. For frames, we used an input image of size $256\times 256\times 3$, with $224\times 224\times 512$ as output. We choose RandAug Cubuk et al. (2020) as strong augmentation, while random cropping, resizing, and random horizontal flip as weak augmentation. We set $\delta$ as 0.6 and $\tau$ as 0.7. More experiments of hyperparameters are shown in Appendix C.5.

We tested DMT’s localization performance with varying amounts of labeled data and found that it consistently outperforms state-of-the-art methods when with 256, 2k, and 4k labeled data. Notably, even with just 256 labeled data, DMT achieved an accuracy of $87.2\%$ to $87.6\%$, showing a significant improvement in CIoU by around 10 absolute points compared to preceding models. Additionally, our model shows a $3\%$ absolute improvement in CIoU compared to a supervised-only model. Furthermore, DMT maintains superior performance in complex and open environments Mo and Morgado (2022b); Ma et al. (2023); Zhu et al. (2024), as demonstrated in Table 2 and Table 3(c), indicating strong generalization capabilities. These results highlight DMT’s ability to improve localization performance by utilizing more unlabeled data.

We categorize objects based on their bounding box pixel area into small ($1\sim 32^{2}$), medium ($32^{2}\sim 96^{2}$), large ($96^{2}\sim 144^{2}$) and huge ($144^{2}\sim 224^{2}$). We tested different methods on small and medium objects in the VGG-Sound dataset, focusing on the challenges of detecting small objects and reducing false positives mentioned earlier. The results in Table 3(a) show that DMT significantly improves performance, especially in terms of MSE metric. Despite some errors in the IoU metric, DMT still outperforms previous methods. The results in Figure 1 show that DMT accurately locates sounding objects with clear boundaries and precisely convergence to object contours, unlike previous methods that often have excessive or insufficient foreground regions, especially for small objects. These results demonstrate the effective and precise localization of small objects achieved by DMT. More experiments of different object sizes are in Appendix C.7.

We present visualized predictions for testsets of varying sizes in Figure 1 (Left). It is evident that our approach achieves remarkable precision in localizing sounding sources. It accurately locates the position of sounding objects and precisely converges to their boundaries, while prior methods usually have excessive or insufficient foreground regions, particularly in the case of small objects.

It is worth noting that our method can even find out sounding objects overlooked in the manual annotations. For instance, in Figure 1 (Left, $4$-th row), the heatmap reveals the presence of a piano, which is omitted in the manual annotation process. Furthermore, we assessed the model’s capability to identify false positives, signifying instances where sounding objects are occasionally not visually observable within the image (off-screen), as shown in 3(b). This reflects the ability of Dual Mean-Teacher to extract audio semantic information and effectively localize multiple sounding objects within a scene, a feat that eludes other methods. We attribute this capability to the semantic alignment of audio-visual features achieved through the pre-trained VGGish and SoundNet backbone.

We tested DMT’s generalization across different domains and its ability to localize multiple objects. Models trained using VGG-ss 144k were directly evaluated on MUSIC-solo Zhao et al. (2018), MUSIC-duet Zhao et al. (2018), and MUSIC-synthetic datasets Hu et al. (2020, 2021). Figure 3 demonstrates DMT’s strong generalization performance in the music domain, outperforming other method. As shown in Figure 3, the previous method struggles to accurately localize multiple sounding objects, either missing them or including all sounding objects within a large foreground area. In contrast, DMT localizes each instrument accurately and separately. However, without category information for fine-grained training, it leads to sub-optimal performance in differentiating between multiple active and silent instruments. There is still significant room for improvement with multiple sounding objects and we plan to address this issue in future work.

In this section, we progressively analyze the performance gain from each module in detail. We choose a self-supervised approach as our baseline. Table 5 presents the results on the Flickr 144k training set with $10\%$ annotated samples.

(i). Initially, we apply it under the semi-supervised pseudo-labeling mechanism, using only one backbone, as shown in Table 5(a) and Table 5(b). The localization performance improves with annotated data supervision, but the gain is limited due to poor-quality pseudo-labels.

(ii) Next, we extend it to a two-backbone architecture and sequentially introduce the filter, IPL, and EMA modules. The results demonstrate that all three modules contribute to performance improvement ($3\%$, $1.8\%$, $1\%$). Notably, Filter shows the most significant impact on the model’s improvement, for it effectively rejects noisy samples, ensuring the stability of the model.

(iii) Finally, we can observe that optimal performance is achieved through the joint integration of the three modules by effectively suppressing confirmation bias.

We aim to demonstrate that even with extremely limited labeled data, significant performance can still be achieved. To this end, we investigated the performance of our model from $0.5\%$ to $10\%$, and report the results in Table 6. Our model consistently outperforms state-of-the-art approaches across all ratios. Furthermore, with a constant amount of unlabeled data, as the proportion of labeled data increases, our model’s performance continues to improve, highlighting the significant impact of labeled data.

We investigated the impact of varying amounts of unlabeled data while keeping labeled data constant. Experimental results in Table 6 show that increasing the amount of unlabeled data improves localization performance, which seems contradictory to the previous conclusion about the proportion of labeled data, but actually, it demonstrates that labeled data can effectively leverage the unlabeled data. Based on our analysis, labeled data not only provides annotation information but also effectively enhances the power of unlabeled data, resulting in significant performance improvements.

Both naive SSL and DMT have utilized labeled and unlabeled data. However, a key distinction is that naive SSL employs unlabeled data only for contrastive loss, whereas DMT leverages pseudo-labels to incorporate unlabeled data into both contrastive loss and supervised loss, which amplifies the utilization of unlabeled data, thus enhancing generalization capability.

Data Utilization. We supplement the comparison experiments with fixed labeled data and an increase in unlabeled data from 10k to 200k, as shown in left part of  7. As the amount of unlabeled data increases, naive SSL exhibits only marginal improvement, whereas DMT shows more performance gains, indicating DMT can better use unlabeled data.

Generalization Ability. The right part of  7 highlights the limitations of naive SSL in the open set and in-the-wild datasets, suggesting that adding a supervised loss alone may lead to overfitting and weaken generalization. In contrast, DMT effectively leverages pseudo-label for improved generalization capability.

We believe that the initialization of teachers and students is crucial, for the quality of pseudo-labels has a significant impact on model performance. To validate the effectiveness of this idea, we experimented to study the warm-up stage’s impact on the model. We find that without the Warm-Up Stage, the model’s improvement is very slow, and the performance eventually deteriorates. This indicates that without a good initialization, the model can accumulate errors, leading to confirmation bias issues. Therefore, we can conclude that Warm-Up is essential as it effectively suppresses confirmation bias in the early stages of training.

In Section 4.3, we present the origins and mitigation strategies for confirmation bias in the localization task. In this section, we will demonstrate this. Figure 4(a) depicts the quality of pseudo labels before and after applying the filter module. It is evident that the quality of pseudo labels can be significantly improved after filtering. This observation highlights the effectiveness of the filter in eliminating noisy samples, which tend to be false-positive instances. Figure 4(b) depicts the comparison between using the direct outputs of each teacher as pseudo labels and utilizing their intersection, known as IPL. Where the purple dashed line represents the initialization value. The results clearly indicate that employing the direct outputs alone leads to the accumulation of bias, causing a deterioration in the quality of pseudo labels throughout the training process. Conversely, IPL consistently ensures the preservation of high-quality pseudo labels, thus mitigating the impact of bias.

Furthermore, Figure 4(c) visually presents the trend of model performance, revealing that the absence of any of these three modules results in a decline in model performance. However, we can see that EMA only affects the final performance of the model, and without the Filter module, the model’s performance will be significantly affected by noise. Without the IPL module, the model will experience a continuous decline in performance due to erroneous estimation of pseudo-labels. Therefore, we find that the Noise Filtering module and IPL modules play a significant role in addressing the confirmation bias problem. Moreover, Figure 4(d) reflects that under the joint action of the three modules, DMT generates more accurate pseudo-labels and its performance continues to improve steadily.