research-article

Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding

Authors:

Chun ChenAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 1631 - 1639

https://doi.org/10.1145/3485447.3512209

Published: 25 April 2022 Publication History

Abstract

Learning low-dimensional representations for Heterogeneous Information Networks (HINs) has drawn increasing attention recently for its effectiveness in real-world applications. Compared with homogeneous networks, HINs are characterized by meta-paths connecting different types of nodes with semantic meanings. Existing methods mainly follow the prototype of independently learning meta-path-based embeddings and integrating them into a unified embedding. However, meta-paths in a HIN are inherently correlated since they reflect different perspectives of the same object. If each meta-path is treated as an isolated semantic data resource and the correlations among them are disregarded, sub-optimality in the both the meta-path based embedding and final embedding will be resulted. To address this issue, we make the first attempt to explicitly model the correlation among meta-paths by proposing Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding (CKD). More specifically, we model the knowledge in each meta-path with two different granularities: regional knowledge and global knowledge. We learn the meta-path-based embeddings by collaboratively distill the knowledge from intra-meta-path and inter-meta-path simultaneously. Experiments conducted on six real-world HIN datasets demonstrates the effectiveness of the CKD method.

References

[1]

Rohan Anil, Gabriel Pereyra, Alexandre Passos, Róbert Ormándi, George E. Dahl, and Geoffrey E. Hinton. 2018. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations.

[2]

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. 2018. Mine: mutual information neural estimation. ArXiv preprint (2018).

[3]

Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019. Representation learning for attributed multiplex heterogeneous network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1358–1368.

Digital Library

[4]

Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. 2020. Online Knowledge Distillation with Diverse Peers. In Proceedings of the AAAI Conference on Artificial Intelligence. 3430–3437.

[5]

Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. 2021. Cross-Layer Distillation with Semantic Calibration. In Proceedings of the AAAI Conference on Artificial Intelligence. 7028–7036.

[6]

Hongxu Chen, Hongzhi Yin, Weiqing Wang, Hao Wang, Quoc Viet Hung Nguyen, and Xue Li. 2018. PME: projected metric embedding on heterogeneous networks for link prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1177–1186.

Digital Library

[7]

Xia Chen, Guoxian Yu, Jun Wang, Carlotta Domeniconi, Zhao Li, and Xiangliang Zhang. 2019. Activehne: Active heterogeneous network embedding. arXiv preprint arXiv:1905.05659(2019).

[8]

Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017. metapath2vec: Scalable representation learning for heterogeneous networks. In SIGKDD. 135–144.

[9]

Yuxiao Dong, Ziniu Hu, Kuansan Wang, Yizhou Sun, and Jie Tang. 2020. Heterogeneous Network Representation Learning. In IJCAI, Vol. 20. 4861–4867.

[10]

Dániel Fogaras, Balázs Rácz, Károly Csalogány, and Tamás Sarlós. 2005. Towards scaling fully personalized pagerank: Algorithms, lower bounds, and experiments. Internet Mathematics(2005), 333–358.

[11]

Tao-yang Fu, Wang-Chien Lee, and Zhen Lei. 2017. Hin2vec: Explore meta-paths in heterogeneous information networks for representation learning. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1797–1806.

[12]

Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020. MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding. In The Web Conference. 2331–2341.

[13]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In NeuraIPS. 1024–1034.

[14]

Yu He, Yangqiu Song, Jianxin Li, Cheng Ji, Jian Peng, and Hao Peng. 2019. Hetespaceywalk: A heterogeneous spacey random walk for heterogeneous information network embedding. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 639–648.

Digital Library

[15]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531(2015).

[16]

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. ArXiv preprint (2018).

[17]

Huiting Hong, Hantao Guo, Yucheng Lin, Xiaoqing Yang, Zang Li, and Jieping Ye. 2020. An attention-based graph neural network for heterogeneous structural learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 4132–4139.

[18]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous graph transformer. In The Web Conference. 2704–2710.

Digital Library

[19]

Xiao Huang, Qingquan Song, Fan Yang, and Xia Hu. 2019. Large-scale heterogeneous feature embedding. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 3878–3885.

Digital Library

[20]

Rana Hussein, Dingqi Yang, and Philippe Cudré-Mauroux. 2018. Are meta-paths necessary? Revisiting heterogeneous graph embeddings. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 437–446.

[21]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. ArXiv preprint (2016).

[22]

Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. ArXiv preprint (2016).

[23]

Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. 2019. Diffusion improves graph learning. In NeuraIPS. 13354–13366.

[24]

Yuanfu Lu, Chuan Shi, Linmei Hu, and Zhiyuan Liu. 2019. Relation structure-aware heterogeneous information network embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 4456–4463.

Digital Library

[25]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781(2013).

[26]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. NeuraIPS (2013), 3111–3119.

[27]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748(2018).

[28]

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. 2020. Graph Representation Learning via Graphical Mutual Information Maximization. In The Web Conference. 259–270.

Digital Library

[29]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 701–710.

Digital Library

[30]

Yuxiang Ren, Bo Liu, Chao Huang, Peng Dai, Liefeng Bo, and Jiawei Zhang. 2019. Heterogeneous deep graph infomax. ArXiv preprint (2019).

[31]

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In Proceedings of the International Conference on Learning Representations.

[32]

Chuan Shi, Yuanfu Lu, Linmei Hu, Zhiyuan Liu, and Huadong Ma. 2020. RHINE: Relation structure-aware heterogeneous information network embedding. IEEE Transactions on Knowledge and Data Engineering (2020).

[33]

Jiaming Song and Stefano Ermon. 2019. Understanding the limitations of variational mutual information estimators. ArXiv preprint (2019).

[34]

Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S Yu, and Tianyi Wu. 2011. Pathsim: Meta path-based top-k similarity search in heterogeneous information networks. VLDB (2011), 992–1003.

Digital Library

[35]

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015. Line: Large-scale information network embedding. In The Web Conference. 1067–1077.

Digital Library

[36]

Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive representation distillation. In Proceedings of the International Conference on Learning Representations.

[37]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. ArXiv preprint (2017).

[38]

Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019. Deep graph infomax. (2019).

[39]

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019. Heterogeneous graph attention network. In The Conference. 2022–2032.

Digital Library

[40]

Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. 2021. Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning. arXiv preprint arXiv:2105.09111(2021).

[41]

Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. 2020. Heterogeneous Network Representation Learning: Survey, Benchmark, Evaluation, and Beyond. TKDE (2020).

[42]

Sergey Zagoruyko and Nikos Komodakis. 2017. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In Proceedings of the International Conference on Learning Representations.

[43]

Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. 2019. Heterogeneous graph neural network. In SIGKDD. 793–803.

[44]

Wentao Zhang, Yuan Fang, Zemin Liu, Min Wu, and Xinming Zhang. 2020. mg2vec: Learning Relationship-Preserving Heterogeneous Graph Representations via Metagraph Embedding. TKDE (2020).

[45]

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4320–4328.

[46]

Jianan Zhao, Xiao Wang, Chuan Shi, Zekuan Liu, and Yanfang Ye. 2020. Network Schema Preserving Heterogeneous Information Network Embedding.

[47]

Sheng Zhou, Jiajun Bu, Xin Wang, Jiawei Chen, and Can Wang. 2019. Hahe: Hierarchical attentive heterogeneous information network embedding. arXiv preprint arXiv:1902.01475(2019).

[48]

Sheng Zhou, Xin Wang, Jiajun Bu, Martin Ester, Pinggang Yu, Jiawei Chen, Qihao Shi, and Can Wang. 2020. DGE: Deep Generative Network Embedding Based on Commonality and Individuality. In AAAI.

[49]

Sheng Zhou, Yucheng Wang, Defang Chen, Jiawei Chen, Xin Wang, Can Wang, and Jiajun Bu. 2021. Distilling Holistic Knowledge with Graph Neural Networks. In International Conference on Computer Vision.

[50]

Sheng Zhou, Hongxia Yang, Xin Wang, Jiajun Bu, Martin Ester, Pinggang Yu, Jianwei Zhang, and Can Wang. 2018. Prre: Personalized relation ranking embedding for attributed networks. In CIKM. 823–832.

Digital Library

Cited By

Zhang HYang XBai LLiang J(2024)Enhancing Drug Recommendations Via Heterogeneous Graph Representation Learning in EHR NetworksIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.332902536:7(3024-3035)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2023.3329025
Chen ZWu ZZhong LPlant CWang SGuo W(2024)Attributed Multi-Order Graph Convolutional Network for Heterogeneous GraphsNeural Networks10.1016/j.neunet.2024.106225174:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106225
Peng LWang XZhu XEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Unsupervised Multiplex Graph learning with Complementary and Consistent InformationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611971(454-462)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611971
Show More Cited By

Index Terms

Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information systems applications

Index terms have been assigned to the content through auto-classification.

Recommendations

Heterogeneous Information Network Embedding with Meta-path Based Graph Attention Networks
Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions
Abstract
Network embedding is an emerging research field which aims at projecting network elements into lower dimensional spaces. However, most network embedding algorithms focus on homogeneous networks, thus cannot be directly applied to the Heterogeneous ...
Easing Embedding Learning by Comprehensive Transcription of Heterogeneous Information Networks
KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Heterogeneous information networks (HINs) are ubiquitous in real-world applications. In the meantime, network embedding has emerged as a convenient tool to mine and learn from networked data. As a result, it is of interest to develop HIN embedding ...
Task-Guided and Path-Augmented Heterogeneous Network Embedding for Author Identification
WSDM '17: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining

In this paper, we study the problem of author identification under double-blind review setting, which is to identify potential authors given information of an anonymized paper. Different from existing approaches that rely heavily on feature engineering, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Natural Science Foundation of China

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
557
Total Downloads

Downloads (Last 12 months)134
Downloads (Last 6 weeks)7

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HYang XBai LLiang J(2024)Enhancing Drug Recommendations Via Heterogeneous Graph Representation Learning in EHR NetworksIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.332902536:7(3024-3035)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2023.3329025
Chen ZWu ZZhong LPlant CWang SGuo W(2024)Attributed Multi-Order Graph Convolutional Network for Heterogeneous GraphsNeural Networks10.1016/j.neunet.2024.106225174:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106225
Peng LWang XZhu XEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Unsupervised Multiplex Graph learning with Complementary and Consistent InformationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611971(454-462)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611971
Liu XLi JWang Y(2023)Research on Battery Energy Control Strategy Based on Neural Network2023 3rd International Conference on Mobile Networks and Wireless Communications (ICMNWC)10.1109/ICMNWC60182.2023.10435801(1-6)Online publication date: 4-Dec-2023
https://doi.org/10.1109/ICMNWC60182.2023.10435801
Li QChen WFang ZYing CWang C(2023)A multi-view contrastive learning for heterogeneous network embeddingScientific Reports10.1038/s41598-023-33324-713:1Online publication date: 25-Apr-2023
https://doi.org/10.1038/s41598-023-33324-7
Wang TPan ZHu GXu KZhang Y(2023)Self-supervised heterogeneous graph learning with iterative similarity distillationKnowledge-Based Systems10.1016/j.knosys.2023.110779276:COnline publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1016/j.knosys.2023.110779
Chen LLi YDeng X(2023)Multi-view learning-based heterogeneous network representation learningJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2023.10185535:10(101855)Online publication date: Dec-2023
https://doi.org/10.1016/j.jksuci.2023.101855
Yan YLi CYu YLi XZhao Z(2023)OSGNNExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.120115225:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.120115
Li CLiu XYan YZhao ZZeng Q(2023)HetGNN-SF: Self-supervised learning on heterogeneous graph neural network via semantic strength and feature similarityApplied Intelligence10.1007/s10489-023-04612-653:19(21902-21919)Online publication date: 16-Jun-2023
https://dl.acm.org/doi/10.1007/s10489-023-04612-6

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents