research-article

A Dual-channel Semi-supervised Learning Framework on Graphs via Knowledge Transfer and Meta-learning

Authors:

Jianqiang Huang,

Xian-Sheng Hua,

Hui XiongAuthors Info & Claims

ACM Transactions on the Web, Volume 18, Issue 2

Article No.: 18, Pages 1 - 26

https://doi.org/10.1145/3577033

Published: 08 January 2024 Publication History

Abstract

This article studies the problem of semi-supervised learning on graphs, which aims to incorporate ubiquitous unlabeled knowledge (e.g., graph topology, node attributes) with few-available labeled knowledge (e.g., node class) to alleviate the scarcity issue of supervised information on node classification. While promising results are achieved, existing works for this problem usually suffer from the poor balance of generalization and fitting ability due to the heavy reliance on labels or task-agnostic unsupervised information. To address the challenge, we propose a dual-channel framework for semi-supervised learning on Graphs via Knowledge Transfer between independent supervised and unsupervised embedding spaces, namely, GKT. Specifically, we devise a dual-channel framework including a supervised model for learning the label probability of nodes and an unsupervised model for extracting information from massive unlabeled graph data. A knowledge transfer head is proposed to bridge the gap between the generalization and fitting capability of the two models. We use the unsupervised information to reconstruct batch-graphs to smooth the label probability distribution on the graphs to improve the generalization of prediction. We also adaptively adjust the reconstructed graphs by encouraging the label-related connections to solidify the fitting ability. Since the optimization of the supervised channel with knowledge transfer contains that of the unsupervised channel as a constraint and vice versa, we then propose a meta-learning-based method to solve the bi-level optimization problem, which avoids the negative transfer and further improves the model’s performance. Finally, extensive experiments validate the effectiveness of our proposed framework by comparing state-of-the-art algorithms.

References

[1]

Firoj Alam, Shafiq Joty, and Muhammad Imran. 2018. Graph-based semi-supervised learning with convolution neural networks to classify crisis related tweets. In 12th International AAAI Conference on Web and Social Media.

[2]

Jiyang Bai, Yuxiang Ren, and Jiawei Zhang. 2020. Ripple walk training: A subgraph-based training framework for large and deep graph neural network. arXiv preprint arXiv:2002.07206 (2020).

[3]

Muthu Balaanand, N. Karthikeyan, S. Karthik, R. Varatharajan, Gunasekaran Manogaran, and C. B. Sivaparthipan. 2019. An enhanced graph-based semi-supervised learning algorithm to detect fake users on Twitter. J. Supercomput. 75, 9 (2019), 6085–6105.

Digital Library

[4]

Jie Chen, Tengfei Ma, and Cao Xiao. 2018. FastGCN: Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations.

[5]

Kaixuan Chen, Lina Yao, Dalin Zhang, Xianzhi Wang, Xiaojun Chang, and Feiping Nie. 2019. A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Trans. Neural Netw. Learn. Syst. 31, 5 (2019), 1747–1756.

[6]

Ming Chen, Zhewei Wei, Bolin Ding, Yaliang Li, Ye Yuan, Xiaoyong Du, and Ji-Rong Wen. 2020. Scalable graph neural networks via bidirectional propagation. Adv. Neural Inf. Process. Syst. 33 (2020), 14556–14566.

[7]

Zhengming Ding, Sheng Li, Ming Shao, and Yun Fu. 2018. Graph adaptive knowledge transfer for unsupervised domain adaptation. In European Conference on Computer Vision (ECCV). 37–52.

Digital Library

[8]

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. PMLR, 1126–1135.

Digital Library

[9]

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. 2017. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning. PMLR, 1165–1173.

[10]

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. 2018. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning. PMLR, 1568–1577.

[11]

Alberto Garcia Duran and Mathias Niepert. 2017. Learning graph representations with embedding propagation. Adv. Neural Inf. Process. Syst. 30 (2017), 5119–5130.

[12]

Sujatha Das Gollapalli, Cornelia Caragea, Prasenjit Mitra, and C. Lee Giles. 2015. Improving researcher homepage classification with unlabeled data. ACM Trans. Web 9, 4 (2015), 1–32.

Digital Library

[13]

William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In 31st International Conference on Neural Information Processing Systems. 1025–1035.

Digital Library

[14]

Xueting Han, Zhenhuan Huang, Bang An, and Jing Bai. 2021. Adaptive transfer learning on graph neural networks. In 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 565–574.

Digital Library

[15]

Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. 2021. Meta-learning in neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (2021). DOI:

[16]

Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. 2021. Meta-learning in neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 44, 9 (2021), 5149–5169.

[17]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020. GPT-GNN: Generative pre-training of graph neural networks. In 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1857–1867.

Digital Library

[18]

Kexin Huang and Marinka Zitnik. 2020. Graph meta learning via local subgraphs. Adv. Neural Inf. Process. Syst. 33 (2020), 5862–5874.

[19]

Yizhu Jiao, Yun Xiong, Jiawei Zhang, Yao Zhang, Tianqi Zhang, and Yangyong Zhu. 2020. Sub-graph contrast for scalable self-supervised graph representation learning. In IEEE International Conference on Data Mining (ICDM). IEEE, 222–231.

[20]

Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).

[21]

Junhyun Lee, Inyeop Lee, and Jaewoo Kang. 2019. Self-attention graph pooling. In International Conference on Machine Learning. PMLR, 3734–3743.

[22]

Junnan Li, Caiming Xiong, and Steven C. H. Hoi. 2021. CoMatch: Semi-supervised learning with contrastive graph regularization. In IEEE/CVF International Conference on Computer Vision. 9475–9484.

[23]

Qimai Li, Xiao-Ming Wu, Han Liu, Xiaotong Zhang, and Zhichao Guan. 2019. Label efficient semi-supervised learning via graph filtering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9582–9591.

[24]

Jiawei Luo, Pingjian Ding, Cheng Liang, and Xiangtao Chen. 2018. Semi-supervised prediction of human miRNA-disease association based on graph regularization framework in heterogeneous networks. Neurocomputing 294 (2018), 29–38.

[25]

Minnan Luo, Xiaojun Chang, Liqiang Nie, Yi Yang, Alexander G. Hauptmann, and Qinghua Zheng. 2017. An adaptive semisupervised feature analysis for video semantic recognition. IEEE Trans. Cyber. 48, 2 (2017), 648–660.

[26]

Qiaozhu Mei, Duo Zhang, and ChengXiang Zhai. 2008. A general optimization framework for smoothing language models on graph structures. In 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 611–618.

Digital Library

[27]

Paul Micaelli and Amos Storkey. 2020. Non-greedy gradient-based hyperparameter optimization over long horizons. arXiv preprint arXiv:2007.07869 (2020).

[28]

Daniel Carlos Guimarães Pedronette, Ying Weng, Alexandro Baldassin, and Chaohuan Hou. 2019. Semi-supervised and active learning through manifold reciprocal kNN graph for image retrieval. Neurocomputing 340 (2019), 19–31.

Digital Library

[29]

Hao Peng, Ruitong Zhang, Yingtong Dou, Renyu Yang, Jingyi Zhang, and Philip S. Yu. 2021. Reinforced neighborhood selection guided multi-relational graph neural networks. ACM Trans. Inf. Syst. 40, 4 (2021), 1–46.

Digital Library

[30]

Hao Peng, Ruitong Zhang, Shaoning Li, Yuwei Cao, Shirui Pan, and Philip Yu. 2022. Reinforced, incremental and cross-lingual event detection from social messages. IEEE Trans. Pattern Anal. Mach. Intell. 45, 1 (2022), 980–998.

[31]

Zhen Peng, Yixiang Dong, Minnan Luo, Xiao-Ming Wu, and Qinghua Zheng. 2020. Self-supervised graph representation learning via global context prediction. arXiv preprint arXiv:2003.01604 (2020).

[32]

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. 2020. Graph representation learning via graphical mutual information maximization. In the Web Conference. 259–270.

Digital Library

[33]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: Online learning of social representations. In 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 701–710.

Digital Library

[34]

Ziyue Qiao, Yi Du, Yanjie Fu, Pengfei Wang, and Yuanchun Zhou. 2019. Unsupervised author disambiguation using heterogeneous graph convolutional network embedding. In IEEE International Conference on Big Data (Big Data). IEEE, 910–919.

[35]

Ziyue Qiao, Yanjie Fu, Pengyang Wang, Meng Xiao, Zhiyuan Ning, Denghui Zhang, Yi Du, and Yuanchun Zhou. 2022. RPT: Toward transferable model on heterogeneous researcher data via pre-training. IEEE Trans. Big Data 9, 1 (2022), 186–199.

[36]

Ziyue Qiao, Pengyang Wang, Yanjie Fu, Yi Du, Pengfei Wang, and Yuanchun Zhou. 2020. Tree structure-aware graph representation learning via integrated hierarchical aggregation and relational metric learning. In IEEE International Conference on Data Mining (ICDM). IEEE, 432–441.

[37]

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020. GCC: Graph contrastive coding for graph neural network pre-training. In 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1150–1160.

Digital Library

[38]

Asmaa Rassil, Hiba Chougrad, and Hamid Zouaki. 2020. The importance of local labels distribution and dominance for node classification in graph neural networks. In 19th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 1505–1511.

[39]

Yuxiang Ren, Jiyang Bai, and Jiawei Zhang. 2021. Label contrastive coding based graph neural network for graph classification. In Database Systems for Advanced Applications, Christian S. Jensen, Ee-Peng Lim, De-Nian Yang, Wang-Chien Lee, Vincent S. Tseng, Vana Kalogeraki, Jen-Wei Huang, and Chih-Ya Shen (Eds.). Springer International Publishing, Cham, 123–140.

[40]

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI Mag. 29, 3 (2008), 93–93.

Digital Library

[41]

Yuanjie Shao, Nong Sang, Changxin Gao, and Li Ma. 2017. Probabilistic class structure regularized sparse representation graph for semi-supervised hyperspectral image classification. Pattern Recog. 63 (2017), 102–114.

Digital Library

[42]

Min Shi, Yufei Tang, Xingquan Zhu, and Jianxun Liu. 2020. Topic-aware web service representation learning. ACM Trans. Web 14, 2 (2020), 1–23.

Digital Library

[43]

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019. Meta-weight-net: Learning an explicit mapping for sample weighting. Adv. Neural Inf. Process. Syst. 32 (2019), 1919–1930.

[44]

Amarnag Subramanya, Slav Petrov, and Fernando Pereira. 2010. Efficient graph-based semi-supervised learning of structured tagging models. In Conference on Empirical Methods in Natural Language Processing. 167–176.

[45]

Fan-Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. 2020. InfoGraph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=r1lfF2NYvH.

[46]

Qiuling Suo, Jingyuan Chou, Weida Zhong, and Aidong Zhang. 2020. TAdaNet: Task-adaptive network for graph-enriched meta-learning. In 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1789–1799.

Digital Library

[47]

Ioannis A. Tamposis, Konstantinos D. Tsirigos, Margarita C. Theodoropoulou, Panagiota I. Kontou, and Pantelis G. Bagos. 2019. Semi-supervised learning of hidden Markov models for biological sequence analysis. Bioinformatics 35, 13 (2019), 2208–2215.

[48]

Zhengzheng Tang, Ziyue Qiao, Xuehai Hong, Yang Wang, Fayaz Ali Dharejo, Yuanchun Zhou, and Yi Du. 2021. Data augmentation for graph convolutional network on semi-supervised classification. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data. Springer, 33–48.

Digital Library

[49]

Yingjie Tian, Mahboubeh Mirzabagheri, Peyman Tirandazi, and Seyed Mojtaba Hosseini Bamakan. 2020. A non-convex semi-supervised approach to opinion spam detection by ramp-one class SVM. Inf. Process. Manag. 57, 6 (2020), 102381.

[50]

Ashwini Tonge and Cornelia Caragea. 2020. Image privacy prediction using deep neural networks. ACM Trans. Web 14, 2 (2020), 1–32.

Digital Library

[51]

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 11 (2008).

[52]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.

[53]

Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. 2018. Deep graph infomax. In International Conference on Learning Representations.

[54]

Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. 2012. Deep learning via semi-supervised embedding. In Neural Networks: Tricks of the Trade. Springer, 639–655.

[55]

Daqing Wu, Xiangyang Guo, Xiao Luo, Ziyue Qiao, and Jinwen Ma. 2022. Adaptive harmony learning and optimization for attributed graph clustering. In International Joint Conference on Neural Networks (IJCNN). IEEE, 1–8.

[56]

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019. Simplifying graph convolutional networks. In International Conference on Machine Learning. PMLR, 6861–6871.

[57]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018).

[58]

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. 2016. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning. PMLR, 40–48.

Digital Library

[59]

Huaxiu Yao, Chuxu Zhang, Ying Wei, Meng Jiang, Suhang Wang, Junzhou Huang, Nitesh Chawla, and Zhenhui Li. 2020. Graph few-shot learning via knowledge transfer. In AAAI Conference on Artificial Intelligence. 6656–6663.

[60]

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. 2020. GraphSAINT: Graph sampling-based inductive learning method. In International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=BJe8pkHFwS.

[61]

Dalin Zhang, Lina Yao, Kaixuan Chen, Sen Wang, Xiaojun Chang, and Yunhao Liu. 2019. Making sense of spatio-temporal preserving representations for EEG-based human intention recognition. IEEE Trans. Cyber. 50, 7 (2019), 3033–3044.

[62]

Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. 2020. Graph-Bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140 (2020).

[63]

Yabin Zhang, Bin Deng, Kui Jia, and Lei Zhang. 2020. Label propagation with augmented anchors: A simple semi-supervised learning baseline for unsupervised domain adaptation. In European Conference on Computer Vision. Springer, 781–797.

Digital Library

[64]

Fan Zhou, Chengtai Cao, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Ji Geng. 2019. Meta-GNN: On few-shot node classification in graph meta-learning. In 28th ACM International Conference on Information and Knowledge Management. 2357–2360.

Digital Library

[65]

Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2021. Meta learning for knowledge distillation. arXiv preprint arXiv:2106.04570 (2021).

[66]

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. 2020. Beyond homophily in graph neural networks: Current limitations and effective designs. Adv. Neural Inf. Process. Syst. 33 (2020).

[67]

Xiaojin Zhu and Zoubin Ghahramani. 2002. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02–107, Carnegie Mellon University.

[68]

Xiaojin Zhu, Zoubin Ghahramani, and John D. Lafferty. 2003. Semi-supervised learning using Gaussian fields and harmonic functions. In 20th International Conference on Machine Learning (ICML’03). 912–919.

Cited By

Qiao ZXiao MGuo WLuo XXiong H(2024)Information filtering and interpolating for semi-supervised graph domain adaptationPattern Recognition10.1016/j.patcog.2024.110498153(110498)Online publication date: Sep-2024
https://doi.org/10.1016/j.patcog.2024.110498
Qiao ZLuo XXiao MDong HZhou YXiong HElkind E(2023)Semi-supervised domain adaptation in graph transfer learningProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/253(2279-2287)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/253
Guo SLiu KWang PDai WDu YZhou YCui W(2023)RDKG: A Reinforcement Learning Framework for Disease Diagnosis on Knowledge Graph2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00122(1049-1054)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00122

Index Terms

A Dual-channel Semi-supervised Learning Framework on Graphs via Knowledge Transfer and Meta-learning
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Multi-task learning
        Transfer learning
      2. Unsupervised learning
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Semi-supervised learning

Recommendations

Improving Semi-Supervised Text Classification with Dual Meta-Learning
The goal of semi-supervised text classification (SSTC) is to train a model by exploring both a small number of labeled data and a large number of unlabeled data, such that the learned semi-supervised classifier performs better than the supervised ...
Semi-supervised partial label learning algorithm via reliable label propagation
Abstract
Partial label learning (PLL) is a weakly supervised learning method that is able to predict one label as the correct answer from a given candidate label set. In PLL, when all possible candidate labels are as signed to real-world training examples, ...
Semi-supervised learning with graphs

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on the Web

ACM Transactions on the Web Volume 18, Issue 2

May 2024

378 pages

EISSN:1559-114X

DOI:10.1145/3613666

Editor:
White Ryen
Microsoft Research, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 January 2024

Online AM: 18 January 2023

Accepted: 20 October 2022

Revised: 16 August 2022

Received: 31 January 2022

Published in TWEB Volume 18, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Foshan HKUST Projects
Natural Science Foundation of China
Strategic Priority Research Program of CAS

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
653
Total Downloads

Downloads (Last 12 months)333
Downloads (Last 6 weeks)35

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Qiao ZXiao MGuo WLuo XXiong H(2024)Information filtering and interpolating for semi-supervised graph domain adaptationPattern Recognition10.1016/j.patcog.2024.110498153(110498)Online publication date: Sep-2024
https://doi.org/10.1016/j.patcog.2024.110498
Qiao ZLuo XXiao MDong HZhou YXiong HElkind E(2023)Semi-supervised domain adaptation in graph transfer learningProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/253(2279-2287)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/253
Guo SLiu KWang PDai WDu YZhou YCui W(2023)RDKG: A Reinforcement Learning Framework for Disease Diagnosis on Knowledge Graph2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00122(1049-1054)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00122

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents