Where and How to Transfer: Knowledge Aggregation-Induced Transferability Perception for Unsupervised Domain Adaptation

Unsupervised domain adaptation without accessing expensive annotation processes of target data has achieved remarkable successes in semantic segmentation. However, most existing state-of-the-art methods cannot explore whether semantic representations across domains are transferable or not, which may result in the negative transfer brought by irrelevant knowledge. To tackle this challenge, in this paper, we develop a novel <underline>K</underline>nowledge <underline>A</underline>ggregation-induced <underline>T</underline>ransferability <underline>P</underline>erception (KATP) module for unsupervised domain adaptation, which is a pioneering attempt to distinguish transferable or untransferable knowledge across domains. Specifically, the KATP module is designed to quantify which semantic knowledge across domains is transferable, by incorporating the transferability information propagation from constructed global category-wise prototypes. Based on KATP, we design a novel KATP Adaptation Network (KATPAN) to determine where and how to transfer. The KATPAN contains a transferable appearance translation module <inline-formula><tex-math notation="LaTeX">$\mathcal {T}_A(\cdot)$</tex-math><alternatives><mml:math><mml:mrow><mml:msub><mml:mi mathvariant="script">T</mml:mi><mml:mi>A</mml:mi></mml:msub><mml:mrow><mml:mo>(</mml:mo><mml:mo>&#x00B7;</mml:mo><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math><inline-graphic xlink:href="dong-ieq1-3128560.gif"/></alternatives></inline-formula> and a transferable representation augmentation module <inline-formula><tex-math notation="LaTeX">$\mathcal {T}_R(\cdot)$</tex-math><alternatives><mml:math><mml:mrow><mml:msub><mml:mi mathvariant="script">T</mml:mi><mml:mi>R</mml:mi></mml:msub><mml:mrow><mml:mo>(</mml:mo><mml:mo>&#x00B7;</mml:mo><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math><inline-graphic xlink:href="dong-ieq2-3128560.gif"/></alternatives></inline-formula>, where both modules construct a virtuous circle of performance promotion. <inline-formula><tex-math notation="LaTeX">$\mathcal {T}_A(\cdot)$</tex-math><alternatives><mml:math><mml:mrow><mml:msub><mml:mi mathvariant="script">T</mml:mi><mml:mi>A</mml:mi></mml:msub><mml:mrow><mml:mo>(</mml:mo><mml:mo>&#x00B7;</mml:mo><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math><inline-graphic xlink:href="dong-ieq3-3128560.gif"/></alternatives></inline-formula> develops a transferability-aware information bottleneck to highlight where to adapt transferable visual characterizations and modality information; <inline-formula><tex-math notation="LaTeX">$\mathcal {T}_R(\cdot)$</tex-math><alternatives><mml:math><mml:mrow><mml:msub><mml:mi mathvariant="script">T</mml:mi><mml:mi>R</mml:mi></mml:msub><mml:mrow><mml:mo>(</mml:mo><mml:mo>&#x00B7;</mml:mo><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math><inline-graphic xlink:href="dong-ieq4-3128560.gif"/></alternatives></inline-formula> explores how to augment transferable representations while abandoning untransferable information, and promotes the translation performance of <inline-formula><tex-math notation="LaTeX">$\mathcal {T}_A(\cdot)$</tex-math><alternatives><mml:math><mml:mrow><mml:msub><mml:mi mathvariant="script">T</mml:mi><mml:mi>A</mml:mi></mml:msub><mml:mrow><mml:mo>(</mml:mo><mml:mo>&#x00B7;</mml:mo><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math><inline-graphic xlink:href="dong-ieq5-3128560.gif"/></alternatives></inline-formula> in return. Comprehensive experiments on several representative benchmark datasets and a medical dataset support the state-of-the-art performance of our model.