research-article

Simultaneous Semantic Alignment Network for Heterogeneous Domain Adaptation

Authors:

Chi Harold Liu,

Zhengming DingAuthors Info & Claims

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Pages 3866 - 3874

https://doi.org/10.1145/3394171.3413995

Published: 12 October 2020 Publication History

Abstract

Heterogeneous domain adaptation (HDA) transfers knowledge across source and target domains that present heterogeneities e.g., distinct domain distributions and difference in feature type or dimension. Most previous HDA methods tackle this problem through learning a domain-invariant feature subspace to reduce the discrepancy between domains. However, the intrinsic semantic properties contained in data are under-explored in such alignment strategy, which is also indispensable to achieve promising adaptability. In this paper, we propose a Simultaneous Semantic Alignment Network (SSAN) to simultaneously exploit correlations among categories and align the centroids for each category across domains. In particular, we propose an implicit semantic correlation loss to transfer the correlation knowledge of source categorical prediction distributions to target domain. Meanwhile, by leveraging target pseudo-labels, a robust triplet-centroid alignment mechanism is explicitly applied to align feature representations for each category. Notably, a pseudo-label refinement procedure with geometric similarity involved is introduced to enhance the target pseudo-label assignment accuracy. Comprehensive experiments on various HDA tasks across text-to-image, image-to-image and text-to-text successfully validate the superiority of our SSAN against state-of-the-art HDA methods. The code is publicly available at https://github.com/BIT-DA/SSAN.

Supplementary Material

MP4 File (3394171.3413995.mp4)

This video introduces a simultaneous semantic alignment network algorithm for heterogeneous domain adaptation, our contribution, experimental results, and published code.

Download
5.36 MB

References

[1]

Massih R. Amini, Nicolas Usunier, and Cyril Goutte. 2009. Learning from Multiple Partially Observed Views -an Application to Multilingual Text Categorization. In NeurIPS. MIT Press, 28--36.

[2]

Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. 2006. Surf: Speeded up robust features. In ECCV. Springer, 404--417.

[3]

Shai Bendavid, John Blitzer, Koby Crammer, and Fernando Pereira. 2006. Analysis of Representations for Domain Adaptation. In NeurIPS. MIT Press, 137--144.

[4]

John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain Adaptation with Structural Correspondence Learning. In EMNLP. ACL, 120--128.

[5]

Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model Compression. In ACM SIGKDD. ACM, 535--541.

[6]

Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. 2019. Progressive Feature Alignment for Unsupervised Domain Adaptation. In CVPR. IEEE, 627--636.

[7]

Weiyu Chen, Tzuming Harry Hsu, Yaohung Hubert Tsai, Yuchiang Frank Wang, and Mingsyan Chen. 2016. Transfer Neural Trees for Heterogeneous Domain Adaptation. In ECCV. Springer, 399--414.

[8]

Tatseng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009. NUS-WIDE: a real-world web image database from National University of Singapore. In CIVR. IEEE, 48.

[9]

Wenyuan Dai, Yuqiang Chen, Guirong Xue, Qiang Yang, and Yong Yu. 2008. Translated Learning: Transfer Learning across Different Feature Spaces. In NeurIPS. MIT Press, 353--360.

[10]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR. IEEE, 248--255.

[11]

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In ICML, Vol. 32. ACM, 647--655.

Digital Library

[12]

Lixin Duan, Dong Xu, and Ivor W Tsang. 2012. Learning with Augmented Features for Heterogeneous Domain Adaptation. In ICML. ACM, 667--674.

[13]

Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. 2013. Unsupervised Visual Domain Adaptation Using Subspace Alignment. In ICCV. IEEE, 2960--2967.

[14]

Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised Domain Adaptation by Backpropagation. In ICML. ACM, 1180--1189.

[15]

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francc ois Laviolette, Mario March, and Victor Lempitsky. 2016. Domain-Adversarial Training of Neural Networks. JMLR, Vol. 17, 59 (2016), 1--35.

Digital Library

[16]

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach. In ICML. ACM, 513--520.

[17]

Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex J Smola. 2007. A kernel method for the two-sample-problem. In NeurIPS. MIT Press, 513--520.

[18]

Gregory Griffin, Alex Holub, and Pietro Perona. 2007. Caltech-256 object category dataset.

[19]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. IEEE, 770--778.

[20]

Geoffrey E Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531 (2015).

[21]

Judy Hoffman, Erik Rodner, Jeff Donahue, Trevor Darrell, and Kate Saenko. 2013. Efficient Learning of Domain-invariant Image Representations. In ICLR.

[22]

Judy Hoffman, Erik Rodner, Jeff Donahue, Brian Kulis, and Kate Saenko. 2014. Asymmetric and Category Invariant Feature Transformations for Domain Adaptation. IJCV, Vol. 109, 1 (2014), 28--41.

Digital Library

[23]

Yuanting Hsieh, Shiyen Tao, Yaohung Hubert Tsai, Yiren Yeh, and Yuchiang Frank Wang. 2016. Recognizing heterogeneous cross-domain data via generalized joint distribution adaptation. In ICME. IEEE, 1--6.

[24]

Brian Kulis, Kate Saenko, and Trevor Darrell. 2011. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms. In CVPR. IEEE, 1785--1792.

[25]

Jingjing Li, Ke Lu, Zi Huang, Lei Zhu, and Hengtao Shen. 2019 b. Heterogeneous Domain Adaptation Through Progressive Alignment. TNNLS, Vol. 30, 5 (2019), 1381--1391.

[26]

Shuang Li, Chi Harold Liu, Qiuxia Lin, Qi Wen, Limin Su, Gao Huang, and Zhengming Ding. 2020 a. Deep Residual Correction Network for Partial Domain Adaptation. TPAMI (2020), 1--1.

[27]

Shuang Li, Chi Harold Liu, Binhui Xie, Limin Su, Zhengming Ding, and Gao Huang. 2019 a. Joint Adversarial Domain Adaptation. In ACM MM. ACM, 729--737.

[28]

Shuang Li, Harold Chi Liu, Qiuxia Lin, Binhui Xie, Zhengming Ding, Gao Huang, and Jian Tang. 2020 b. Domain Conditioned Adaptation Network. In AAAI. AAAI Press, 11386--11393.

[29]

Shuang Li, Shiji Song, Gao Huang, Zhengming Ding, and Cheng Wu. 2018. Domain Invariant and Class Discriminative Feature Learning for Visual Domain Adaptation. TIP, Vol. 27, 9 (2018), 4260--4273.

[30]

Wen Li, Lixin Duan, Dong Xu, and Ivor W Tsang. 2014. Learning With Augmented Features for Supervised and Semi-Supervised Heterogeneous Domain Adaptation. TPAMI, Vol. 36, 6 (2014), 1134--1148.

Digital Library

[31]

Mingsheng Long, Yue Cao, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. 2018a. Transferable Representation Learning with Deep Adaptation Networks. TPAMI (2018).

[32]

Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. 2018b. Conditional adversarial domain adaptation. In NeurIPS. MIT Press, 1647--1657.

[33]

Zelun Luo, Yuliang Zou, Judy Hoffman, and Fei-Fei Li. 2017. Label Efficient Learning of Transferable Representations acrosss Domains and Tasks. In NeurIPS. MIT Press, 165--177.

[34]

Andrew L. Maas. 2013. Rectifier Nonlinearities Improve Neural Network Acoustic Models.

[35]

Liqiang Nie, Wenjie Wang, Richang Hong, Meng Wang, and Qi Tian. 2019. Multimodal Dialog System: Generating Responses via Adaptive Decoders. In ACM MM. ACM, 1098--1106.

[36]

Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. 2011. Domain adaptation via transfer component analysis. TNNLS, Vol. 22, 2 (2011), 199--210.

Digital Library

[37]

Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. TKDE, Vol. 22, 10 (2010), 1345--1359.

Digital Library

[38]

Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-Wah Ngo, and Tao Mei. 2019. Transferrable Prototypical Networks for Unsupervised Domain Adaptation. In CVPR. IEEE, 2239--2247.

[39]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS. MIT Press, 8024--8035.

[40]

Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert R.G. Lanckriet, Roger Levy, and Nuno Vasconcelos. 2010. A New Approach to Cross-Modal Multimedia Retrieval. In ACM MM. ACM, 251--260.

[41]

Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. 2010. Adapting visual category models to new domains. In ECCV. Springer, 213--226.

[42]

Xiaoxiao Shi, Qi Liu, Wei Fan, Philip S Yu, and Ruixin Zhu. 2010. Transfer Learning on Heterogenous Feature Spaces via Spectral Transformation. In ICDE. IEEE, 1049--1054.

[43]

Xiangbo Shu, Guo-Jun Qi, Jinhui Tang, and Jingdong Wang. 2015. Weakly-Shared Deep Transfer Networks for Heterogeneous-Domain Knowledge Propagation. In ACM MM. ACM, 35--44.

[44]

Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical Networks for Few-shot Learning. In NeurIPS. MIT Press, 4077--4087.

[45]

Tatiana Tommasi and Tinne Tuytelaars. 2014. A testbed for cross-dataset analysis. In ECCV. Springer, 18--31.

[46]

Yaohung Hubert Tsai, Yiren Yeh, and Yuchiang Frank Wang. 2016a. Learning Cross-Domain Landmarks for Heterogeneous Domain Adaptation. In CVPR. IEEE, 5081--5090.

[47]

Yao-Huang Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang. 2016b. Heterogeneous domain adaptation with label and structure consistency. In ICASSP. IEEE, 2842--2846.

[48]

Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. 2015. Simultaneous deep transfer across domains and tasks. In ICCV. IEEE, 4068--4076.

[49]

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In CVPR. IEEE, 2962--2971.

[50]

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014. Deep domain confusion: Maximizing for domain invariance. arXiv:1412.3474 (2014).

[51]

Chang Wang and Sridhar Mahadevan. 2011. Heterogeneous domain adaptation using manifold alignment. In IJCAI. Morgan Kaufmann, 1541--1546.

[52]

Xinxiao Wu, Han Wang, Cuiwei Liu, and Yunde Jia. 2013. Cross-View Action Recognition over Heterogeneous Feature Spaces. In ICCV. IEEE, 609--616.

[53]

Min Xiao and Yuhong Guo. 2015a. Feature Space Independent Semi-Supervised Domain Adaptation via Kernel Matching. TPAMI, Vol. 37, 1 (2015), 54--66.

[54]

Min Xiao and Yuhong Guo. 2015b. Semi-supervised subspace co-projection for multi-class heterogeneous domain adaptation. In ECML. Springer, 525--540.

[55]

Shaoan Xie and Zibin Zheng. 2018. Learning Semantic Representations for Unsupervised Domain Adaptation. In ICML. ACM, 5419--5428.

[56]

Yuguang Yan, Wen Li, Hanrui Wu, Huaqing Min, Mingkui Tan, and Qingyao Wu. 2018. Semi-Supervised Optimal Transport for Heterogeneous Domain Adaptation. In IJCAI. Morgan Kaufmann, 2969--2975.

[57]

Yuan Yao, Yu Zhang, Xutao Li, and Yunming Ye. 2019. Heterogeneous Domain Adaptation via Soft Transfer Network. In ACM MM. ACM, 1578--1586.

[58]

Han-Jia Ye, Xiang-Rong Sheng, De-Chuan Zhan, and Peng He. 2018. Distance Metric Facilitated Transportation between Heterogeneous Domains. In IJCAI. Morgan Kaufmann, 3012--3018.

[59]

Joey Tianyi Zhou, Ivor W Tsang, Sinno Jialin Pan, and Mingkui Tan. 2014. Heterogeneous Domain Adaptation for Multiple Classes. In AISTATS. JMLR, 1095--1103.

[60]

Junyan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In ICCV. IEEE, 2242--2251.

[61]

Junbao Zhuo, Shuhui Wang, Weigang Zhang, and Qingming Huang. 2017. Deep Unsupervised Convolutional Domain Adaptation. In ACM MM. ACM, 261--269.

Cited By

Wu JDai HKent KYen JXu CWang Y(2024)Open Set Dandelion Network for IoT Intrusion DetectionACM Transactions on Internet Technology10.1145/363982224:1(1-26)Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1145/3639822
Chen YMancini MZhu XAkata Z(2024)Semi-Supervised and Unsupervised Deep Visual Learning: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.320157646:3(1327-1347)Online publication date: Mar-2024
https://doi.org/10.1109/TPAMI.2022.3201576
Lu YLin DWen JShen LLi XWen Z(2024)Heterogeneous domain adaptation via incremental discriminative knowledge consistencyPattern Recognition10.1016/j.patcog.2024.110857(110857)Online publication date: Jul-2024
https://doi.org/10.1016/j.patcog.2024.110857
Show More Cited By

Index Terms

Simultaneous Semantic Alignment Network for Heterogeneous Domain Adaptation
1. Computing methodologies
  1. Machine learning

Recommendations

Heterogeneous Domain Adaptation via Soft Transfer Network
MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Heterogeneous domain adaptation (HDA) aims to facilitate the learning task in a target domain by borrowing knowledge from a heterogeneous source domain. In this paper, we propose a Soft Transfer Network (STN), which jointly learns a domain-shared ...
Knowledge Preserving and Distribution Alignment for Heterogeneous Domain Adaptation
Domain adaptation aims at improving the performance of learning tasks in a target domain by leveraging the knowledge extracted from a source domain. To this end, one can perform knowledge transfer between these two domains. However, this problem becomes ...
Cross-domain structure preserving projection for heterogeneous domain adaptation
Highlights
- Extending locality preserving projection (LPP) to multi-domain scenarios.
- ...
Abstract
Heterogeneous Domain Adaptation (HDA) addresses the transfer learning problems where data from the source and target domains are of different modalities (e.g., texts and images) or feature dimensions (e.g., features extracted with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

October 2020

4889 pages

ISBN:9781450379885

DOI:10.1145/3394171

General Chairs:
Chang Wen Chen
Chinese University of Hong Kong, Shenzhen, China
,
Rita Cucchiara
UNIMORE, Italy
,
Xian-Sheng Hua
Alibaba Group, China
,
Program Chairs:
Guo-Jun Qi
Futurewei Technologies, USA
,
Elisa Ricci
UNITN & Fondazione Bruno Kessler, Italy
,
Zhengyou Zhang
Tencent, China
,
Roger Zimmermann
National University of Singapore, Singapore

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12 - 16, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
299
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)6

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu JDai HKent KYen JXu CWang Y(2024)Open Set Dandelion Network for IoT Intrusion DetectionACM Transactions on Internet Technology10.1145/363982224:1(1-26)Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1145/3639822
Chen YMancini MZhu XAkata Z(2024)Semi-Supervised and Unsupervised Deep Visual Learning: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.320157646:3(1327-1347)Online publication date: Mar-2024
https://doi.org/10.1109/TPAMI.2022.3201576
Lu YLin DWen JShen LLi XWen Z(2024)Heterogeneous domain adaptation via incremental discriminative knowledge consistencyPattern Recognition10.1016/j.patcog.2024.110857(110857)Online publication date: Jul-2024
https://doi.org/10.1016/j.patcog.2024.110857
Ye XWang K(2024)Deep generative domain adaptation with temporal relation attention mechanism for cross-user activity recognitionPattern Recognition10.1016/j.patcog.2024.110811156(110811)Online publication date: Dec-2024
https://doi.org/10.1016/j.patcog.2024.110811
Wang WLi ZLi W(2024)Graph embedding-based heterogeneous domain adaptation with domain-invariant feature learning and distributional order preservingNeural Networks10.1016/j.neunet.2023.11.048170(427-440)Online publication date: Feb-2024
https://doi.org/10.1016/j.neunet.2023.11.048
Li XZheng YMa HQi ZMeng XMeng L(2024)Cross-modal learning using privileged information for long-tailed image classificationComputational Visual Media10.1007/s41095-023-0382-0Online publication date: 10-Jun-2024
https://doi.org/10.1007/s41095-023-0382-0
Wu LWang HGong LYao YGuo XLi B(2024)Multi-modal Domain Adaptation Method Based on Parameter Fusion and Two-Step AlignmentNeural Processing Letters10.1007/s11063-024-11567-356:2Online publication date: 15-Mar-2024
https://doi.org/10.1007/s11063-024-11567-3
Wu HWu YLi NYang MZhang JNg MLong J(2024)High-order proximity and relation analysis for cross-network heterogeneous node classificationMachine Learning10.1007/s10994-024-06566-3113:9(6247-6272)Online publication date: 19-Jun-2024
https://doi.org/10.1007/s10994-024-06566-3
Zheng YLi ZLi XLiu JWang YMeng XMeng L(2024)Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal AlignmentArtificial Neural Networks and Machine Learning – ICANN 202410.1007/978-3-031-72347-6_8(110-125)Online publication date: 17-Sep-2024
https://doi.org/10.1007/978-3-031-72347-6_8
Ye XWang K(2024)Cross-User Activity Recognition via Temporal Relation Optimal TransportMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63989-0_18(355-374)Online publication date: 19-Jul-2024
https://doi.org/10.1007/978-3-031-63989-0_18
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents