Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Knowledge Preserving and Distribution Alignment for Heterogeneous Domain Adaptation

Published: 08 September 2021 Publication History

Abstract

Domain adaptation aims at improving the performance of learning tasks in a target domain by leveraging the knowledge extracted from a source domain. To this end, one can perform knowledge transfer between these two domains. However, this problem becomes extremely challenging when the data of these two domains are characterized by different types of features, i.e., the feature spaces of the source and target domains are different, which is referred to as heterogeneous domain adaptation (HDA). To solve this problem, we propose a novel model called Knowledge Preserving and Distribution Alignment (KPDA), which learns an augmented target space by jointly minimizing information loss and maximizing domain distribution alignment. Specifically, we seek to discover a latent space, where the knowledge is preserved by exploiting the Laplacian graph terms and reconstruction regularizations. Moreover, we adopt the Maximum Mean Discrepancy to align the distributions of the source and target domains in the latent space. Mathematically, KPDA is formulated as a minimization problem with orthogonal constraints, which involves two projection variables. Then, we develop an algorithm based on the Gauss–Seidel iteration scheme and split the problem into two subproblems, which are solved by searching algorithms based on the Barzilai–Borwein (BB) stepsize. Promising results demonstrate the effectiveness of the proposed method.

References

[1]
P. A. Absil, Robert Mahony, and Rodolphe Sepulchre. 2009. Optimization Algorithms on Matrix Manifolds. Princeton University Press.
[2]
Massih Amini, Nicolas Usunier, and Cyril Goutte. 2009. Learning from multiple partially observed views-an application to multilingual text categorization. In Proceedings of the Advances in Neural Information Processing Systems. 28–36.
[3]
Jonathan Barzilai and Jonathan M. Borwein. 1988. Two-point step size gradient methods. IMA Journal of Numerical Analysis 8, 1 (1988), 141–148.
[4]
Mikhail Belkin and Partha Niyogi. 2002. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Proceedings of the Advances in Neural Information Processing Systems. 585–591.
[5]
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics.
[6]
Jérôme Bolte, Shoham Sabach, and Marc Teboulle. 2014. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Mathematical Programming 146, 1–2 (2014), 459–494.
[7]
Deng Cai, Xiaofei He, and Jiawei Han. 2007. Spectral regression: A unified approach for sparse subspace learning. In Proceedings of the IEEE International Conference on Data Mining. IEEE, 73–82.
[8]
Deng Cai, Xiaofei He, Xuanhui Wang, Hujun Bao, and Jiawei Han. 2009. Locality preserving nonnegative matrix factorization. In Proceedings of the International Joint Conferences on Artificial Intelligence. 1010–1015.
[9]
Manliang Cao, Xiangdong Zhou, Yiming Xu, Yue Pang, and Bo Yao. 2019. Adversarial domain adaptation with semantic consistency for cross-domain image classification. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 259–268.
[10]
Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 3 (2011), 1–27.
[11]
Wei-Yu Chen, Tzu-Ming Harry Hsu, Yao-Hung Hubert Tsai, Ming-Syan Chen, and Yu-Chiang Frank Wang. 2019. Transfer neural trees: Semi-supervised heterogeneous domain adaptation and beyond. IEEE Transactions on Image Processing 28, 9 (2019), 4620–4633.
[12]
Wei-Yu Chen, Tzu-Ming Harry Hsu, Yao-Hung Hubert Tsai, Yu-Chiang Frank Wang, and Ming-Syan Chen. 2016. Transfer neural trees for heterogeneous domain adaptation. In Proceedings of the European Conference on Computer Vision. Springer, 399–414.
[13]
Zhihong Chen, Rong Xiao, Chenliang Li, Gangfeng Ye, Haochuan Sun, and Hongbo Deng. 2020. ESAM: Discriminative domain adaptation with non-displayed items to improve long-tail performance. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
[14]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning 20, 3 (1995), 273–297.
[15]
Gabriela Csurka. 2017. A comprehensive survey on domain adaptation for visual applications. Domain Adaptation in Computer Vision Applications. Springer.
[16]
Hal Daume III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 256–263.
[17]
Zhengming Ding and Yun Fu. 2018. Deep transfer low-rank coding for cross-domain learning. IEEE Transactions on Neural Networks and Learning Systems 30, 6 (2018), 1768–1779.
[18]
Andrea Esuli, Alejandro Moreo, and Fabrizio Sebastiani. 2019. Funnelling: A new ensemble method for heterogeneous transfer learning and its application to cross-lingual text classification. ACM Transactions on Information Systems 37, 3 (2019), 1–30.
[19]
Roger Fletcher. 2013. Practical Methods of Optimization. John Wiley & Sons.
[20]
Alex Gammerman, Volodya Vovk, and Vladimir Vapnik. 2013. Learning by transduction. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence. 148–155.
[21]
Donald Goldfarb, Zaiwen Wen, and Wotao Yin. 2009. A curvilinear search method for p-harmonic flows on spheres. SIAM Journal on Imaging Sciences 2, 1 (2009), 84–109.
[22]
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. 2012. Geodesic flow kernel for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2066–2073.
[23]
Xiaofei He, Shuicheng Yan, Yuxiao Hu, Partha Niyogi, and Hong-Jiang Zhang. 2005. Face recognition using laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 3 (2005), 328–340.
[24]
Xiaofei He, Shuicheng Yan, Yuxiao Hu, and Hong-Jiang Zhang. 2003. Learning a locality preserving subspace for visual recognition. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 385–392.
[25]
Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang. 2016. Learning cross-domain landmarks for heterogeneous domain adaptation. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 5081–5090.
[26]
Thorsten Joachims. 1999. Svmlight: Support vector machine. SVM-Light Support Vector Machine. Retrieved July 22, 2021 from https://www.cs.cornell.edu/people/tj/svm_light/.
[27]
Phillip Keung, Yichao Lu, György Szarvas, and Noah A Smith. 2020. The multilingual amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 4563–4568.
[28]
Elyor Kodirov, Tao Xiang, and Shaogang Gong. 2017. Semantic autoencoder for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3174–3183.
[29]
Haoliang Li, Sinno Jialin Pan, Renjie Wan, and Alex C. Kot. 2019. Heterogeneous transfer learning via deep matrix completion with adversarial kernel embedding. In Proceedings of the International Joint Conference on Artificial Intelligence.
[30]
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. 2020. Heterogeneous domain adaptation via nonlinear matrix factorization. IEEE Transactions on Neural Networks and Learning Systems 31, 3 (2020), 984–996.
[31]
Jingjing Li, Erpeng Chen, Zhengming Ding, Lei Zhu, Ke Lu, and Heng Tao Shen. 2020. Maximum density divergence for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).
[32]
Jingjing Li, Mengmeng Jing, Ke Lu, Lei Zhu, and Heng Tao Shen. 2019. Locality preserving joint transfer for domain adaptation. IEEE Transactions on Image Processing 28, 12 (2019), 6103–6115.
[33]
Jingjing Li, Mengmeng Jing, Hongzu Su, Ke Lu, Lei Zhu, and Heng Tao Shen. 2021. Faster domain adaptation networks. IEEE Transactions on Knowledge and Data Engineering (2021).
[34]
Jingjing Li, Ke Lu, Zi Huang, Lei Zhu, and Heng Tao Shen. 2019. Heterogeneous domain adaptation through progressive alignment. IEEE Transactions on Neural Networks and Learning Systems 30, 5 (2019), 1381–1391.
[35]
Shuang Li, Shiji Song, and Gao Huang. 2017. Prediction reweighting for domain adaptation. IEEE Transactions on Neural Networks and Learning Systems 28, 7 (2017), 1682–1695.
[36]
Wen Li, Lixin Duan, Dong Xu, and Ivor W. Tsang. 2014. Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 6 (2014), 1134–1148.
[37]
Xukun Li and Doina Caragea. 2020. Domain adaptation with reconstruction for disaster tweet classification. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1561–1564.
[38]
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. 2013. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 2200–2207.
[39]
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579–2605.
[40]
Jorge Nocedal and Stephen Wright. 2006. Numerical Optimization. Springer Science & Business Media.
[41]
Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345–1359.
[42]
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. 2018. Multi-adversarial domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence.
[43]
Peter Prettenhofer and Benno Stein. 2010. Cross-language text classification using structural correspondence learning. In Proceedings of the Association of Computational Linguistics.
[44]
Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the 1st Instructional Conference on Machine Learning, Vol. 242. 133–142.
[45]
Lixin Su, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2020. Continual domain adaptation for machine reading comprehension. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management.
[46]
Ben Tan, Yangqiu Song, Erheng Zhong, and Qiang Yang. 2015. Transitive transfer learning. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1155–1164.
[47]
Wei Wang, Baopu Li, Shuhui Yang, Jing Sun, Zhengming Ding, Junyang Chen, Xiao Dong, Zhihui Wang, and Haojie Li. 2021. A unified joint maximum mean discrepancy for domain adaptation. arXiv:2101.09979. Retrieved from https://arxiv.org/abs/2101.09979
[48]
Zaiwen Wen and Wotao Yin. 2013. A feasible method for optimization with orthogonality constraints. Mathematical Programming 142, 1–2 (2013), 397–434.
[49]
Hanrui Wu, Yuguang Yan, Sentao Chen, Xiangkang Huang, Qingyao Wu, and Michael K. Ng. 2021. Joint visual and semantic optimization for zero-shot learning. Knowledge-Based Systems 215, 2 (2021), 106773.
[50]
Hanrui Wu, Yuguang Yan, Guosheng Lin, Min Yang, Michael K. Ng, and Qingyao Wu. 2020. Iterative refinement for multi-source visual domain adaptation. IEEE Transactions on Knowledge and Data Engineering.
[51]
Hanrui Wu, Yuguang Yan, Michael K Ng, and Qingyao Wu. 2020. Domain-attention conditional wasserstein distance for multi-source domain adaptation. ACM Transactions on Intelligent Systems and Technology 11, 4 (2020), 1–19.
[52]
Hanrui Wu, Yuguang Yan, Yuzhong Ye, Huaqing Min, Michael K. Ng, and Qingyao Wu. 2019. Online heterogeneous transfer learning by knowledge transition. ACM Transactions on Intelligent Systems and Technology 10, 3 (2019), 1–19.
[53]
Hanrui Wu, Yuguang Yan, Yuzhong Ye, Michael K. Ng, and Qingyao Wu. 2020. Geometric knowledge embedding for unsupervised domain adaptation. Knowledge-Based Systems 191, 1 (2020), 105155.
[54]
Qingyao Wu, Michael K. Ng, and Yunming Ye. 2014. Cotransfer learning using coupled markov chains with restart. IEEE Intelligent Systems 29, 4 (2014), 26–33.
[55]
Qingyao Wu, Hanrui Wu, Xiaoming Zhou, Mingkui Tan, Yonghui Xu, Yuguang Yan, and Tianyong Hao. 2017. Online transfer learning with multiple homogeneous or heterogeneous sources. IEEE Transactions on Knowledge and Data Engineering 29, 7 (2017), 1494–1507.
[56]
Qingyao Wu, Xiaoming Zhou, Yuguang Yan, Hanrui Wu, and Huaqing Min. 2017. Online transfer learning by leveraging multiple source domains. Knowledge and Information Systems 52, 3 (2017), 687–707.
[57]
Min Xiao and Yuhong Guo. 2014. Feature space independent semi-supervised domain adaptation via kernel matching. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 1 (2014), 54–66.
[58]
Min Xiao and Yuhong Guo. 2015. Semi-supervised subspace co-projection for multi-class heterogeneous domain adaptation. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 525–540.
[59]
Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo. 2017. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2272–2281.
[60]
Shuicheng Yan, Dong Xu, Benyu Zhang, Hong-Jiang Zhang, Qiang Yang, and Stephen Lin. 2006. Graph embedding and extensions: A general framework for dimensionality reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 1 (2006), 40–51.
[61]
Yuguang Yan, Wen Li, Michael KP Ng, Mingkui Tan, Hanrui Wu, Huaqing Min, and Qingyao Wu. 2017. Learning discriminative correlation subspace for heterogeneous domain adaptation. In Proceedings of the International Joint Conference on Artificial Intelligence. 3252–3258.
[62]
Yuguang Yan, Wen Li, Hanrui Wu, Huaqing Min, Mingkui Tan, and Qingyao Wu. 2018. Semi-supervised optimal transport for heterogeneous domain adaptation. In Proceedings of the International Joint Conference on Artificial Intelligence. 2969–2975.
[63]
Yuguang Yan, Hanrui Wu, Yuzhong Ye, Chaoyang Bi, Min Lu, Dapeng Liu, Qingyao Wu, and Michael Kwok-Po Ng. 2021. Transferable feature selection for unsupervised domain adaptation. IEEE Transactions on Knowledge and Data Engineering.
[64]
Liu Yang, Liping Jing, Jian Yu, and Michael K Ng. 2015. Learning transferred weights from co-occurrence data for heterogeneous transfer learning. IEEE Transactions on Neural Networks and Learning Systems 27, 11 (2015), 2187–2200.
[65]
Yuan Yao, Yu Zhang, Xutao Li, and Yunming Ye. 2020. Discriminative distribution alignment: A unified framework for heterogeneous domain adaptation. Pattern Recognition 101, 10 (2020), 107165.
[66]
Hongchao Zhang and William W Hager. 2004. A nonmonotone line search technique and its application to unconstrained optimization. SIAM Journal on Optimization 14, 4 (2004), 1043–1056.
[67]
Wen Zhang and Dongrui Wu. 2020. Discriminative joint probability maximum mean discrepancy (DJP-MMD) for domain adaptation. In Proceedings of the International Joint Conference on Neural Networks. IEEE, 1–8.
[68]
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. 2019. Bridging theory and algorithm for domain adaptation. In Proceedings of the International Conference on Machine Learning. 7404–7413.
[69]
Joey Tianyi Zhou, Sinno Jialin Pan, and Ivor W. Tsang. 2019. A deep learning framework for hybrid heterogeneous transfer learning. Artificial Intelligence 275 (2019), 310–328.
[70]
Bin Zhu, Chong-Wah Ngo, and Jing-jing Chen. 2020. Cross-domain cross-modal food transfer. In Proceedings of the 28th ACM International Conference on Multimedia. 3762–3770.
[71]
Yin Zhu, Yuqiang Chen, Zhongqi Lu, Sinno Jialin Pan, Gui-Rong Xue, Yong Yu, and Qiang Yang. 2011. Heterogeneous transfer learning for image classification. In Proceedings of the AAAI Conference on Artificial Intelligence. 1304–1309.

Cited By

View all
  • (2024)Improving Semi-Supervised Text Classification with Dual Meta-LearningACM Transactions on Information Systems10.1145/364861242:4(1-28)Online publication date: 26-Apr-2024
  • (2024)Robust Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661380(3009-3012)Online publication date: 10-Jul-2024
  • (2024)Are Large Language Models Good at Utility Judgments?Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657784(1941-1951)Online publication date: 10-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems
ACM Transactions on Information Systems  Volume 40, Issue 1
January 2022
599 pages
ISSN:1046-8188
EISSN:1558-2868
DOI:10.1145/3483337
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2021
Accepted: 01 June 2021
Revised: 01 March 2021
Received: 01 September 2020
Published in TOIS Volume 40, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Heterogeneous domain adaptation
  2. transfer learning
  3. domain adaptation
  4. local structure
  5. reconstruction
  6. distribution alignment

Qualifiers

  • Research-article
  • Refereed

Funding Sources

  • HKRGC GRF
  • HKU-TCL Joint Research Centre for Artificial Intelligence
  • National Natural Science Foundation of China (NSFC)
  • Key-Area Research and Development Program of Guangdong Province

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)80
  • Downloads (Last 6 weeks)7
Reflects downloads up to 28 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Improving Semi-Supervised Text Classification with Dual Meta-LearningACM Transactions on Information Systems10.1145/364861242:4(1-28)Online publication date: 26-Apr-2024
  • (2024)Robust Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661380(3009-3012)Online publication date: 10-Jul-2024
  • (2024)Are Large Language Models Good at Utility Judgments?Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657784(1941-1951)Online publication date: 10-Jul-2024
  • (2024)Online Transfer Learning for RSV Case Detection2024 IEEE 12th International Conference on Healthcare Informatics (ICHI)10.1109/ICHI61247.2024.00074(512-521)Online publication date: 3-Jun-2024
  • (2024)Unsupervised heterogeneous domain adaptation for EEG classificationJournal of Neural Engineering10.1088/1741-2552/ad5fbd21:4(046018)Online publication date: 16-Jul-2024
  • (2024)Domain adaptation with contrastive and adversarial oriented transferable semantic augmentationKnowledge-Based Systems10.1016/j.knosys.2023.111092282:COnline publication date: 27-Feb-2024
  • (2024)Distributed Heterogeneous Transfer LearningBig Data Research10.1016/j.bdr.2024.10045637:COnline publication date: 28-Aug-2024
  • (2024)Semi-supervised multi-source transfer learning for cross-subject EEG motor imagery classificationMedical & Biological Engineering & Computing10.1007/s11517-024-03032-z62:6(1655-1672)Online publication date: 7-Feb-2024
  • (2024)Heterogeneous transfer learning: recent developments, applications, and challengesMultimedia Tools and Applications10.1007/s11042-024-18352-383:27(69759-69795)Online publication date: 2-Feb-2024
  • (2024)High-order proximity and relation analysis for cross-network heterogeneous node classificationMachine Language10.1007/s10994-024-06566-3113:9(6247-6272)Online publication date: 1-Sep-2024
  • Show More Cited By

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media