research-article

Graph Self-supervised Learning with Augmentation-aware Contrastive Learning

Authors:

Weidong XiaoAuthors Info & Claims

WWW '23: Proceedings of the ACM Web Conference 2023

Pages 154 - 164

https://doi.org/10.1145/3543507.3583246

Published: 30 April 2023 Publication History

Abstract

Graph self-supervised learning aims to mine useful information from unlabeled graph data, and has been successfully applied to pre-train graph representations. Many existing approaches use contrastive learning to learn powerful embeddings by learning contrastively from two augmented graph views. However, none of these graph contrastive methods fully exploits the diversity of different augmentations, and hence is prone to overfitting and limited generalization ability of learned representations. In this paper, we propose a novel Graph Self-supervised Learning method with Augmentation-aware Contrastive Learning. Our method is based on the finding that the pre-trained model after adding augmentation diversity can achieve better generalization ability. To make full use of the information from the diverse augmentation method, this paper constructs new augmentation-aware prediction task which complementary with the contrastive learning task. Similar to how pre-training requires fast adaptation to different downstream tasks, we simulate train-test adaptation on the constructed tasks for further enhancing the learning ability; this strategy can be deemed as a form of meta-learning. Experimental results show that our method outperforms previous methods and learns better representations for a variety of downstream tasks.

Supplemental Material

PDF File

Appendix

Download
428.67 KB

References

[1]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In ICML(Proceedings of Machine Learning Research, Vol. 119). PMLR, 1597–1607.

[2]

Austin Derrow-Pinion, Jennifer She, David Wong, Oliver Lange, Todd Hester, Luis Perez, Marc Nunkesser, Seongjae Lee, Xueying Guo, Brett Wiltshire, Peter W. Battaglia, Vishal Gupta, Ang Li, Zhongwen Xu, Alvaro Sanchez-Gonzalez, Yujia Li, and Petar Velickovic. 2021. ETA Prediction with Graph Neural Networks in Google Maps. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, 3767–3776. https://doi.org/10.1145/3459637.3481916

[3]

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In ICML(Proceedings of Machine Learning Research, Vol. 70). PMLR, 1126–1135.

[4]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS(JMLR Proceedings, Vol. 9). JMLR.org, 249–256.

[5]

Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proceedings. 2005 IEEE international joint conference on neural networks, Vol. 2. 729–734.

[6]

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

[7]

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In NeurIPS.

[8]

Kaveh Hassani and Amir Hosein Khas Ahmadi. 2020. Contrastive Multi-View Representation Learning on Graphs. In ICML(Proceedings of Machine Learning Research, Vol. 119). PMLR, 4116–4126.

[9]

Kaveh Hassani and Amir Hosein Khasahmadi. 2020. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning. PMLR, 4116–4126.

[10]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729–9738.

[11]

R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In ICLR. OpenReview.net.

[12]

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurIPS.

[13]

Kexin Huang and Marinka Zitnik. 2020. Graph Meta Learning via Local Subgraphs. In NeurIPS.

[14]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR (Poster).

[15]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).

[16]

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR (Poster). OpenReview.net.

[17]

Namkyeong Lee, Junseok Lee, and Chanyoung Park. 2021. Augmentation-free self-supervised learning on graphs. arXiv preprint arXiv:2112.02472 (2021).

[18]

Yuanze Lin, Xun Guo, and Yan Lu. 2021. Self-Supervised Video Representation Learning with Meta-Contrastive Network. In ICCV. IEEE, 8219–8229.

[19]

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. 2020. Graph Representation Learning via Graphical Mutual Information Maximization. In WWW. ACM / IW3C2, 259–270.

[20]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learning of social representations. In KDD. ACM, 701–710.

[21]

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020. GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. In KDD. ACM, 1150–1160.

[22]

Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences 63, 10 (2020), 1872–1897.

[23]

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2019. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXiv:1907.10903 (2019).

[24]

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veličković, and Michal Valko. 2022. Large-Scale Representation Learning on Graphs via Bootstrapping. In International Conference on Learning Representations.

[25]

Puja Trivedi, Ekdeep Singh Lubana, Yujun Yan, Yaoqing Yang, and Danai Koutra. 2022. Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practices. In WWW. ACM, 1538–1549.

[26]

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. 2019. Deep Graph Infomax. In ICLR (Poster). OpenReview.net.

[27]

Hao Wang, Tong Xu, Qi Liu, Defu Lian, Enhong Chen, Dongfang Du, Han Wu, and Wen Su. 2019. MCNE: an end-to-end framework for learning multiple conditional network representations of social network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1064–1072.

[28]

Ning Wang, Minnan Luo, Kaize Ding, Lingling Zhang, Jundong Li, and Qinghua Zheng. 2020. Graph few-shot learning with attribute matching. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 1545–1554.

[29]

Yaochen Xie, Zhao Xu, Zhengyang Wang, and Shuiwang Ji. 2021. Self-Supervised Learning of Graph Neural Networks: A Unified Review. CoRR abs/2102.10757 (2021).

[30]

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems 33 (2020), 5812–5823.

[31]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 (2020).

[32]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep Graph Contrastive Representation Learning. CoRR abs/2006.04131 (2020).

[33]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep Graph Contrastive Representation Learning. CoRR abs/2006.04131 (2020).

[34]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2021. Graph Contrastive Learning with Adaptive Augmentation. In WWW. ACM / IW3C2, 2069–2080.

[35]

Marinka Zitnik and Jure Leskovec. 2017. Predicting multicellular function through multi-layer tissue networks. Bioinform. 33, 14 (2017), i190–i198. https://doi.org/10.1093/bioinformatics/btx252

Cited By

Tan SLi DJiang RZhang YOkumura MSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Community-invariant graph contrastive learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694008(47579-47606)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694008
Huang YLu WYang YLarson K(2024)An efficient prototype-based clustering approach for edge pruning in graph neural networks to battle over-smoothingProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/464(4201-4209)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/464
Chen DZhao XXiao W(2024)Fine-Grained Anomaly Detection on Dynamic Graphs via Attention Alignment2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00246(3178-3190)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00246

Index Terms

Graph Self-supervised Learning with Augmentation-aware Contrastive Learning
1. Information systems
  1. World Wide Web
    1. Web mining

Recommendations

Self-supervised Graph-level Representation Learning with Adversarial Contrastive Learning
The recently developed unsupervised graph representation learning approaches apply contrastive learning into graph-structured data and achieve promising performance. However, these methods mainly focus on graph augmentation for positive samples, while the ...
JGCL: Joint Self-Supervised and Supervised Graph Contrastive Learning
WWW '22: Companion Proceedings of the Web Conference 2022

Semi-supervised and self-supervised learning on graphs are two popular avenues for graph representation learning. We demonstrate that no single method from semi-supervised and self-supervised learning works uniformly well for all settings in the node ...
Generalized self-supervised contrastive learning with bregman divergence for image recognition
Highlights
- A generalized contrastive learning framework is proposed.
- Bregman divergence ...
Abstract
Contrastive learning techniques continue to receive a lot of attention in the self-supervised learning area. Specifically, the learned distance features can be further utilized to capture the distance between latent features in the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '23: Proceedings of the ACM Web Conference 2023

April 2023

4293 pages

ISBN:9781450394161

DOI:10.1145/3543507

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Data Availability

Appendix https://dl.acm.org/doi/10.1145/3543507.3583246#Appendix.pdf

Funding Sources

NSFC
HKUST(GZ)
GZU-HKUST

Conference

WWW '23

Sponsor:

SIGWEB

WWW '23: The ACM Web Conference 2023

April 30 - May 4, 2023

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
1,106
Total Downloads

Downloads (Last 12 months)283
Downloads (Last 6 weeks)20

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tan SLi DJiang RZhang YOkumura MSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Community-invariant graph contrastive learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694008(47579-47606)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694008
Huang YLu WYang YLarson K(2024)An efficient prototype-based clustering approach for edge pruning in graph neural networks to battle over-smoothingProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/464(4201-4209)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/464
Chen DZhao XXiao W(2024)Fine-Grained Anomaly Detection on Dynamic Graphs via Attention Alignment2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00246(3178-3190)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00246

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents