research-article

DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks

Authors:

Aravind Sankar,

Hao YangAuthors Info & Claims

WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

Pages 519 - 527

https://doi.org/10.1145/3336191.3371845

Published: 22 January 2020 Publication History

Abstract

Learning node representations in graphs is important for many applications such as link prediction, node classification, and community detection. Existing graph representation learning methods primarily target static graphs while many real-world graphs evolve over time. Complex time-varying graph structures make it challenging to learn informative node representations over time.

We present Dynamic Self-Attention Network (DySAT), a novel neural architecture that learns node representations to capture dynamic graph structural evolution. Specifically, DySAT computes node representations through joint self-attention along the two dimensions of structural neighborhood and temporal dynamics. Compared with state-of-the-art recurrent methods modeling graph evolution, dynamic self-attention is efficient, while achieving consistently superior performance. We conduct link prediction experiments on two graph types: communication networks and bipartite rating networks. Experimental results demonstrate significant performance gains for DySAT over several state-of-the-art graph embedding baselines, in both single and multi-step link prediction tasks. Furthermore, our ablation study validates the effectiveness of jointly modeling structural and temporal self-attention.

References

[1]

Mart'in Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et almbox. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations (ICLR) .

[3]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems. 3844--3852.

[4]

Lun Du, Yun Wang, Guojie Song, Zhicong Lu, and Junshan Wang. 2018. Dynamic Network Embedding: An Extended Approach for Skip-gram based Network Embedding. In IJCAI . 2086--2092.

[5]

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 1243--1252.

Digital Library

[6]

Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. 2018. dyngraph2vec: Capturing network dynamics using dynamic graph representation learning. arXiv preprint arXiv:1809.02657 (2018).

[7]

Palash Goyal, Nitin Kamra, Xinran He, and Yan Liu. 2017. DynGEM: Deep Embedding Method for Dynamic Graphs. In IJCAI International Workshop on Representation Learning for Graphs (ReLiG) .

[8]

Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 855--864.

Digital Library

[9]

Ehsan Hajiramezanali, Arman Hasanzadeh, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. 2019. Variational graph recurrent neural networks. In Advances in Neural Information Processing Systems. 10700--10710.

[10]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems. 1024--1034.

[11]

F Maxwell Harper and Joseph A Konstan. 2016. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TIIS), Vol. 5, 4 (2016), 19.

[12]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) .

[13]

Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference for Learning Representations (ICLR) .

[14]

Bryan Klimt and Yiming Yang. 2004. Introducing the Enron Corpus. In CEAS 2004 - First Conference on Email and Anti-Spam, July 30--31, 2004, Mountain View, California, USA.

[15]

Adit Krishnan, Hari Cheruvu, Cheng Tao, and Hari Sundaram. 2019. A Modular Adversarial Approach to Social Recommendation. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. ACM, 1753--1762.

Digital Library

[16]

Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. 2007. Graph evolution: Densification and shrinking diameters. ACM Transactions on Knowledge Discovery from Data (TKDD), Vol. 1, 1 (2007), 2.

[17]

Jundong Li, Harsh Dani, Xia Hu, Jiliang Tang, Yi Chang, and Huan Liu. 2017. Attributed network embedding for learning in a dynamic environment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 387--396.

Digital Library

[18]

Xi Liu, Ping-Chun Hsieh, Nick Duffield, Rui Chen, Muhe Xie, and Xidao Wen. 2018. Streaming Network Embedding through Local Actions. arXiv preprint arXiv:1811.05932 (2018).

[19]

Kanika Narang, Chaoqi Yang, Adit Krishnan, Junting Wang, Hari Sundaram, and Carolyn Sutter. 2019. An Induced Multi-Relational Framework for Answer Selection in Community Question Answer Platforms. arXiv preprint arXiv:1911.06957 (2019).

[20]

Giang Hoang Nguyen, John Boaz Lee, Ryan A Rossi, Nesreen K Ahmed, Eunyee Koh, and Sungchul Kim. 2018. Continuous-time dynamic network embeddings. In 3rd International Workshop on Learning Representations for Big Networks (WWW BigNet) .

Digital Library

[21]

Pietro Panzarasa, Tore Opsahl, and Kathleen M. Carley. 2009. Patterns and dynamics of users' behavior and interaction: Network analysis of an online community. JASIST, Vol. 60, 5 (2009), 911--932.

[22]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . ACM, 701--710.

Digital Library

[23]

Aravind Sankar, Adit Krishnan, Zongjian He, and Carl Yang. 2019 a. Rase: Relationship aware social embedding. In 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.

[24]

Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. 2018. Dynamic Graph Representation Learning via Self-Attention Networks. arXiv preprint arXiv:1812.09430 (2018).

[25]

Aravind Sankar, Xinyang Zhang, and Kevin Chen-Chuan Chang. 2017. Motif-based Convolutional Neural Network on Graphs. CoRR, Vol. abs/1711.05697 (2017). arxiv: 1711.05697

[26]

Aravind Sankar, Xinyang Zhang, and Kevin Chen-Chuan Chang. 2019 b. Meta-GNN: Metagraph Neural Network for Semi-supervised learning in Attributed Heterogeneous Information Networks. In 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 137--144.

Digital Library

[27]

Purnamrita Sarkar and Andrew W Moore. 2006. Dynamic social network analysis using latent space models. In Advances in Neural Information Processing Systems. 1145--1152.

[28]

Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In Thirty-Second AAAI Conference on Artificial Intelligence .

[29]

Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. In Thirty-Second AAAI Conference on Artificial Intelligence .

[30]

Joshua B Tenenbaum, Vin De Silva, and John C Langford. 2000. A global geometric framework for nonlinear dimensionality reduction. science, Vol. 290, 5500 (2000), 2319--2323.

[31]

Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 3462--3471.

Digital Library

[32]

Rakshit Trivedi, Mehrdad Farajtbar, Prasenjeet Biswal, and Hongyuan Zha. 2018. Representation Learning over Dynamic Graphs. arXiv preprint arXiv:1803.04051 (2018).

[33]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems. 6000--6010.

[34]

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations (ICLR) .

[35]

Petar Velivc ković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2018. Deep graph infomax. arXiv preprint arXiv:1809.10341 (2018).

[36]

Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Twenty-Eighth AAAI conference on artificial intelligence .

Digital Library

[37]

Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. In International Conference on Learning Representations (ICLR) .

[38]

Ziwei Zhang, Peng Cui, Jian Pei, Xiao Wang, and Wenwu Zhu. 2018. Timers: Error-bounded svd restart on dynamic networks. In Thirty-Second AAAI Conference on Artificial Intelligence .

[39]

Lekui Zhou, Yang Yang, Xiang Ren, Fei Wu, and Yueting Zhuang. 2018. Dynamic network embedding by modeling triadic closure process. In Thirty-Second AAAI Conference on Artificial Intelligence .

[40]

Linhong Zhu, Dong Guo, Junming Yin, Greg Ver Steeg, and Aram Galstyan. 2016. Scalable Temporal Latent Space Inference for Link Prediction in Dynamic Social Networks. IEEE Trans. Knowl. Data Eng., Vol. 28, 10 (2016), 2765--2777.

Digital Library

[41]

Marinka Zitnik, Monica Agrawal, and Jure Leskovec. 2018. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, Vol. 34, 13 (2018), 457--466.

[42]

Yuan Zuo, Guannan Liu, Hao Lin, Jia Guo, Xiaoqian Hu, and Junjie Wu. 2018. Embedding Temporal Network via Neighborhood Formation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2857--2866.

Digital Library

Cited By

Liao ZLiu THe YLin L(2024)Effective Temporal Graph Learning via Personalized PageRankEntropy10.3390/e2607058826:7(588)Online publication date: 10-Jul-2024
https://doi.org/10.3390/e26070588
Zhu YGuo JLi HLiu SLi Y(2024)TSAGNN: Temporal link predict method based on two stream adaptive graph neural networkIntelligent Data Analysis10.3233/IDA-23736728:1(77-97)Online publication date: 3-Feb-2024
https://doi.org/10.3233/IDA-237367
Li HDi SLi CChen LZhou X(2024)Fight Fire with Fire: Towards Robust Graph Neural Networks on Dynamic Graphs via Actively DefenseProceedings of the VLDB Endowment10.14778/3659437.365945717:8(2050-2063)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.14778/3659437.3659457
Show More Cited By

Index Terms

DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Information systems
  1. World Wide Web
    1. Web applications
      1. Social networks

Recommendations

Dynamic Network Embedding in Hyperbolic Space via Self-attention
Web Engineering
Abstract
Graph Neural Networks (GNNs) have recently become increasingly popular due to their ability to learn node representations in complex graphs. Existing graph representation learning methods primarily target static graphs in Euclidean space, while ...
TE-DyGE: Temporal Evolution-Enhanced Dynamic Graph Embedding Network
Database Systems for Advanced Applications
Abstract
Dynamic graph representation has gradually attached more attention in the research area since many real-world graphs evolve over time. Recent works typically represent the dynamic graph as a sequence of discrete static subgraphs and apply a ...
DynGCN: A Dynamic Graph Convolutional Network Based on Spatial-Temporal Modeling
Web Information Systems Engineering – WISE 2020
Abstract
Representation learning on graphs has recently attracted a lot of interest with graph convolutional networks (GCN) achieving state-of-the-art performance in many graph mining tasks. However, most of existing methods mainly focus on static graphs ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

January 2020

950 pages

ISBN:9781450368223

DOI:10.1145/3336191

General Chairs:
James Caverlee
Texas A&M University
,
Xia "Ben" Hu
Texas A&M University
,
Program Chairs:
Mounia Lalmas
Spotify
,
Wei Wang
University of California, Los Angeles

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 January 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WSDM '20

Sponsor:

WSDM '20: The Thirteenth ACM International Conference on Web Search and Data Mining

February 3 - 7, 2020

TX, Houston, USA

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

300
Total Citations
View Citations
3,062
Total Downloads

Downloads (Last 12 months)711
Downloads (Last 6 weeks)80

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liao ZLiu THe YLin L(2024)Effective Temporal Graph Learning via Personalized PageRankEntropy10.3390/e2607058826:7(588)Online publication date: 10-Jul-2024
https://doi.org/10.3390/e26070588
Zhu YGuo JLi HLiu SLi Y(2024)TSAGNN: Temporal link predict method based on two stream adaptive graph neural networkIntelligent Data Analysis10.3233/IDA-23736728:1(77-97)Online publication date: 3-Feb-2024
https://doi.org/10.3233/IDA-237367
Li HDi SLi CChen LZhou X(2024)Fight Fire with Fire: Towards Robust Graph Neural Networks on Dynamic Graphs via Actively DefenseProceedings of the VLDB Endowment10.14778/3659437.365945717:8(2050-2063)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.14778/3659437.3659457
Chen CGao DZhang YWang QFu ZZhang XZhu JGu YYu G(2024)NeutronStream: A Dynamic GNN Training Framework with Sliding Window for Graph StreamsProceedings of the VLDB Endowment10.14778/3632093.363210817:3(455-468)Online publication date: 20-Jan-2024
https://doi.org/10.14778/3632093.3632108
Zhong XZhang JHua QYang LGao Z(2024)Short-Term Origin-Destination Demand Prediction Based on Spatiotemporal Encoder-Decoder Network with a Residual Feature ExtractorTransportation Research Record: Journal of the Transportation Research Board10.1177/03611981241248160Online publication date: 24-Jul-2024
https://doi.org/10.1177/03611981241248160
Ang GLim E(2024)Learning Dynamic Multimodal Network Slot Concepts from the Web for Forecasting Environmental, Social and Governance RatingsACM Transactions on the Web10.1145/366367418:3(1-32)Online publication date: 3-May-2024
https://dl.acm.org/doi/10.1145/3663674
Gupta SBedathur S(2024)Robust Training of Temporal GNNs using Nearest Neighbour based Hard NegativesProceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD)10.1145/3632410.3632464(54-63)Online publication date: 4-Jan-2024
https://dl.acm.org/doi/10.1145/3632410.3632464
Wang RHuang JZhang YLi JWang YZhao WLiu SMendis CAbdelzaher THui Yang GWang HHan SHauff CZuccon GZhang Y(2024)TGOnline: Enhancing Temporal Graph Learning with Adaptive Online Meta-LearningProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657791(1659-1669)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657791
Wang FZhu GYuan CHuang YHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)LLM-enhanced Cascaded Multi-level Learning on Temporal Heterogeneous GraphsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657731(512-521)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657731
Wang YMendis CTsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)TGLite: A Lightweight Programming Framework for Continuous-Time Temporal Graph Neural NetworksProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3620665.3640414(1183-1199)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620665.3640414
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents