research-article

DNformer: Temporal Link Prediction with Transfer Learning in Dynamic Networks

Authors:

Tomas WardAuthors Info & Claims

ACM Transactions on Knowledge Discovery from Data, Volume 17, Issue 3

Article No.: 43, Pages 1 - 21

https://doi.org/10.1145/3551892

Published: 22 February 2023 Publication History

Abstract

Temporal link prediction (TLP) is among the most important graph learning tasks, capable of predicting dynamic, time-varying links within networks. The key problem of TLP is how to explore potential link-evolving tendency from the increasing number of links over time. There exist three major challenges toward solving this problem: temporal nonlinear sparsity, weak serial correlation, and discontinuous structural dynamics. In this article, we propose a novel transfer learning model, called DNformer, to predict temporal link sequence in dynamic networks. The structural dynamic evolution is sequenced into consecutive links one by one over time to inhibit temporal nonlinear sparsity. The self-attention of the model is used to capture the serial correlation between the input and output link sequences. Moreover, our structural encoding is designed to obtain changing structures from the consecutive links and to learn the mapping between link sequences. This structural encoding consists of two parts: the node clustering encoding of each link and the link similarity encoding between links. These encodings enable the model to perceive the importance and correlation of links. Furthermore, we introduce a measurement of structural similarity in the loss function for the structural differences of link sequences. The experimental results demonstrate that our model outperforms other state-of-the-art TLP methods such as Transformer, TGAT, and EvolveGCN. It achieves the three highest AUC and four highest precision scores in five different representative dynamic networks problems.

References

[1]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv e-prints, arXiv: stat.ML/1607.06450.

[2]

Deng Cai and Wai Lam. 2020. Graph transformer for graph-to-sequence learning. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 7464–7471.

[3]

Jinyin Chen, Jian Zhang, Xuanheng Xu, Chenbo Fu, Dan Zhang, Qingpeng Zhang, and Qi Xuan. 2021. E-LSTM-D: A deep learning framework for dynamic network link prediction. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51, 6 (2021), 3699–3712. DOI:

[4]

Jia Chen, Ming Zhong, Jianxin Li, Dianhui Wang, Tieyun Qian, and Hang Tu. 2022. Effective deep attributed network representation learning with topology adapted smoothing. IEEE Transactions on Cybernetics 52, 7 (2022), 5935–5946. DOI:

[5]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. DOI:

[6]

Lun Du, Yun Wang, Guojie Song, Zhicong Lu, and Junshan Wang. 2018. Dynamic network embedding: An extended approach for skip-gram based network embedding. In Proceedings of the 27th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 2086–2092.

[7]

Daniel M. Dunlavy, Tamara G. Kolda, and Evrim Acar. 2011. Temporal link prediction using matrix and tensor factorizations. ACM Transactions on Knowledge Discovery from Data 5, 2, Article 10 (Feb.2011), 27 pages. DOI:

Digital Library

[8]

Vijay Prakash Dwivedi and Xavier Bresson. 2020. A generalization of transformer networks to graphs. arXiv e-prints, arXiv:2012.09699.

[9]

Xiaofeng Gao, Zhenhao Cao, Sha Li, Bin Yao, Guihai Chen, and Shaojie Tang. 2019. Taxonomy and evaluation for microblog popularity prediction. ACM Transactions on Knowledge Discovery from Data 13, 2, Article 15 (March2019), 40 pages. DOI:

Digital Library

[10]

Anmol Gulati, James Qin, ChungCheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, and Yonghui Wu. 2020. Conformer: Convolution-augmented transformer for speech recognition. In Proceedings of INTERSPEECH. International Speech Communication Association, Shanghai, China, 5036–5040.

[11]

Chengbin Hou, Han Zhang, Shan He, and Ke Tang. 2020. GloDyNE: Global topology preserving dynamic network embedding. IEEE Transactions on Knowledge and Data Engineering (2020), 1–1. DOI:

Digital Library

[12]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous graph transformer. In Proceedings of the Web Conference 2020. Association for Computing Machinery, New York, NY, 2704–2710. DOI:

Digital Library

[13]

Jin Huang, Feiping Nie, Heng Huang, Yi-Cheng Tu, and Yu Lei. 2013. Social trust prediction using heterogeneous networks. ACM Transactions on Knowledge Discovery from Data 7, 4, Article 17 (Dec.2013), 21 pages. DOI:

Digital Library

[14]

Ruthwik R. Junuthula, Kevin S. Xu, and Vijay K. Devabhaktuni. 2016. Evaluating link prediction accuracy in dynamic networks with added and removed edges. In Proceedings of 2016 IEEE International Conferences on Big Data and Cloud Computing. IEEE Computer Society, Los Alamitos, CA, 377–384. DOI:

[15]

David Kempe, Jon Kleinberg, and Éva Tardos. 2003. Maximizing the spread of influence through a social network. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, 137–146. DOI:

Digital Library

[16]

Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Minneapolis, Minnesota, 2284–2293. DOI:

[17]

Xiaoyi Li, Nan Du, Hui Li, Kang Li, Jing Gao, and Aidong Zhang. 2014. A deep learning approach to link prediction in dynamic networks. In Proceedings of the 2014 SIAM International Conference on Data Mining, Vol. 1. Society for Industrisl and Applied Mathematics, Philadelphia, Pennsylvania, 289–297.

[18]

David Liben-Nowell and Jon Kleinberg. 2007. The link-prediction problem for social networks. Journal of the American Society for Information Science and Technology 58, 7 (2007), 1019–1031. DOI:

Digital Library

[19]

Xutong Liu, Feng Chen, Yen-Cheng Lu, and Chang-Tien Lu. 2017. Spatial prediction for multivariate non-Gaussian data. ACM Transactions on Knowledge Discovery from Data 11, 3, Article 36 (March2017), 27 pages. DOI:

Digital Library

[20]

Xuanqing Liu, Hsiang-Fu Yu, Inderjit S. Dhillon, and Cho-Jui Hsieh. 2020. Learning to encode position for transformer with continuous dynamical model. In Proceedings of the 37th International Conference on Machine Learning. JMLR.org, Vienna, AUSTRIA, Article 587, 9 pages.

[21]

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the2021 IEEE/CVF International Conference on Computer Vision. Computer Vision Foundation, Montreal, Canada, 9992–10002. DOI:

[22]

Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao B. Schardl, and Charles E. Leiserson. 2020. EvolveGCN: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 5679–5681.

[23]

Sylvain Paris, Samuel W. Hasinoff, and Jan Kautz. 2015. Local laplacian filters: Edge-aware image processing with a Laplacian pyramid. Communications of the ACM 58, 3 (Feb.2015), 81–91. DOI:

Digital Library

[24]

Hao Peng, Jianxin Li, Hao Yan, Gong Qiran, Senzhang Wang, Lin Liu, Lihong Wang, and Xiang Ren. 2020. Dynamic network embedding via incremental skip-gram with negative sampling. Science China Information Sciences 63, 10 (2020), 19. DOI:

[25]

Hao Peng, Renyu Yang, Zheng Wang, Jianxin Li, Lifang He, Philip S. Yu, Albert Y. Zomaya, and Rajiv Ranjan. 2022. Lime: Low-cost and incremental learning for dynamic heterogeneous information networks. IEEE Transactions on Computers 71, 3 (2022), 628–642. DOI:

Digital Library

[26]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, 701–710.

Digital Library

[27]

Cunlai Pu, Jie Li, Jian Wang, and Tony Q. S. Quek. 2022. The node-similarity distribution of complex networks and its applications in link prediction. IEEE Transactions on Knowledge and Data Engineering 34, 8 (2022), 4011–4023. DOI:

[28]

Zhenyu Qiu, Jia Wu, Wenbin Hu, Bo Du, Guocai Yuan, and Philip Yu. 2021. Temporal link prediction with motifs for social networks. IEEE Transactions on Knowledge and Data Engineering (2021), 1–1. DOI:

[29]

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. 2020. Self-supervised graph transformer on large-scale molecular data. In Proceedings of Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., Vancouver, Canada, 12559–12571.

[30]

Andrea Rossi, Denilson Barbosa, Donatella Firmani, Antonio Matinata, and Paolo Merialdo. 2021. Knowledge graph embedding for link prediction: A comparative analysis. ACM Transactions on Knowledge Discovery from Data 15, 2, Article 14 (Jan.2021), 49 pages. DOI:

Digital Library

[31]

Ryan A. Rossi and Nesreen K. Ahmed. 2015. The network data repository with interactive graph analytics and visualization. In Proceedings of AAAI Conference on Artificial Intelligence.AAAI Press, Palo Alto, CA, 4292–4293. Retrieved from http://networkrepository.com.

[32]

Umang Sharan and Jennifer Neville. 2008. Temporal-relational classifiers for prediction in evolving domains. In Proceedings of 8th IEEE international conference on data mining. IEEE Computer Society, Los Alamitos, CA, 540–549.

Digital Library

[33]

Laura M. Smith, Linhong Zhu, Kristina Lerman, and Allon G. Percus. 2016. Partitioning networks with node attributes by compressing information flow. ACM Transactions on Knowledge Discovery from Data 11, 2 (2016), 1–26. DOI:

Digital Library

[34]

Shichang Sun, Hongbo Liu, Jiana Meng, C. L. Philip Chen, and Yu Yang. 2018. Substructural regularization with data-sensitive granularity for sequence transfer learning. IEEE Transactions on Neural Networks and Learning Systems 29, 6 (2018), 2545–2557. DOI:

[35]

Yizhou Sun, Jie Tang, Jiawei Han, Cheng Chen, and Manish Gupta. 2014. Co-evolution of multi-typed objects in dynamic star networks. IEEE Transactions on Knowledge and Data Engineering 26, 12 (2014), 2942–2955. DOI:

[36]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, 6000–6010.

Digital Library

[37]

Huan Wang, Wenbin Hu, Zhenyu Qiu, and Bo Du. 2017. Nodes’ evolution diversity and link prediction in social networks. IEEE Transactions on Knowledge and Data Engineering 29, 10 (2017), 2263–2274. DOI:

Digital Library

[38]

Stanley Wasserman and Katherine Faust. 1994. Social Network Analysis: Methods and Applications. Cambridge University Press, Cambridge, UK. DOI:

[39]

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. 2021. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems 32, 1 (2021), 4–24. DOI:

[40]

Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2020. Inductive representation learning on temporal graphs. In Proceedings of International Conference on Learning Representations. OpenReview.net, Ithaca, NY, 1.

[41]

Jin Xu, Yubo Tao, Yuyu Yan, and Hai Lin. 2020. Exploring evolution of dynamic networks via diachronic node embeddings. IEEE Transactions on Visualization and Computer Graphics 26, 7 (2020), 2387–2402. DOI:

[42]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How powerful are graph neural networks? In Proceedings of International Conference on Learning Representations. OpenReview.net, Ithaca, NY, 1.

[43]

Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the 32th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 7444–7452.

[44]

Min Yang, Junhao Liu, Lei Chen, Zhou Zhao, Xiaojun Chen, and Ying Shen. 2020. An advanced deep generative framework for temporal link prediction in dynamic networks. IEEE Transactions on Cybernetics 50, 12 (2020), 4946–4957.

[45]

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021. Do transformers really perform badly for graph representation? In Proceedings of Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., Vancouver, Canada, 28877–28888.

[46]

Wenchao Yu, Charu C. Aggarwal, and Wei Wang. 2017. Temporally factorized network modeling for evolutionary network analysis. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, 455–464.

Digital Library

[47]

Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, 5171–5181.

Digital Library

[48]

Lekui Zhou, Yang Yang, Xiang Ren, Fei Wu, and Yueting Zhuang. 2018. Dynamic network embedding by modeling triadic closure process. In Proceedings of the 32th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 571–578.

[49]

Linhong Zhu, Dong Guo, Junming Yin, Greg Ver Steeg, and Aram Galstyan. 2016. Scalable temporal latent space inference for link prediction in dynamic social networks. IEEE Transactions on Knowledge and Data Engineering 28, 10 (2016), 2765–2777.

Digital Library

Cited By

Li MLiu KLiu HZhao ZWard TWu X(2024)Heterogeneous Meta-Path Graph Learning for Higher-Order Social RecommendationACM Transactions on Knowledge Discovery from Data10.1145/367365818:8(1-25)Online publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1145/3673658
Gao TDaamen WKrishnakumari PHoogendoorn S(2024)Map‐matching for cycling travel data in urban areaIET Intelligent Transport Systems10.1049/itr2.12567Online publication date: 19-Sep-2024
https://doi.org/10.1049/itr2.12567
Jiang XLiu HYang LZhang BWard TSnášel V(2024)Unraveling human social behavior motivations via inverse reinforcement learning-based link predictionComputing10.1007/s00607-024-01279-w106:6(1963-1986)Online publication date: 2-Apr-2024
https://dl.acm.org/doi/10.1007/s00607-024-01279-w
Show More Cited By

Index Terms

DNformer: Temporal Link Prediction with Transfer Learning in Dynamic Networks
1. Information systems
  1. Information systems applications
    1. Data mining
      1. Clustering
2. Theory of computation
  1. Design and analysis of algorithms
    1. Graph algorithms analysis
  2. Theory and algorithms for application domains
    1. Machine learning theory
      1. Unsupervised learning and clustering

Recommendations

Few-shot Link Prediction in Dynamic Networks
WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining

Dynamic link prediction, which aims at forecasting future edges of a node in a dynamic network, is an important problem in network science and has a wide range of real-world applications. A key property of dynamic networks is that new nodes and links ...
A deep contrastive framework for unsupervised temporal link prediction in dynamic networks
Abstract
In dynamic networks, temporal link prediction aims to predict the appearance and disappearance of links in future snapshots based on the network structure we have observed. It also plays a crucial role in network analysis and predicting the ...
A supervised link prediction method for dynamic networks

Link prediction is an important sub-task in link mining area. This paper discusses link prediction in dynamic networks and proposes a new link prediction method which can learn from the long-term graph evolution of networks. The method first represents ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data

ACM Transactions on Knowledge Discovery from Data Volume 17, Issue 3

April 2023

379 pages

ISSN:1556-4681

EISSN:1556-472X

DOI:10.1145/3583064

Editor:
Charu Aggarwal
IBM T. J. Watson Research, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 February 2023

Online AM: 02 August 2022

Accepted: 20 July 2022

Revised: 17 March 2022

Received: 23 November 2021

Published in TKDD Volume 17, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Science Foundation Ireland
Liaoning Collaborative Fund

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
1,140
Total Downloads

Downloads (Last 12 months)315
Downloads (Last 6 weeks)34

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li MLiu KLiu HZhao ZWard TWu X(2024)Heterogeneous Meta-Path Graph Learning for Higher-Order Social RecommendationACM Transactions on Knowledge Discovery from Data10.1145/367365818:8(1-25)Online publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1145/3673658
Gao TDaamen WKrishnakumari PHoogendoorn S(2024)Map‐matching for cycling travel data in urban areaIET Intelligent Transport Systems10.1049/itr2.12567Online publication date: 19-Sep-2024
https://doi.org/10.1049/itr2.12567
Jiang XLiu HYang LZhang BWard TSnášel V(2024)Unraveling human social behavior motivations via inverse reinforcement learning-based link predictionComputing10.1007/s00607-024-01279-w106:6(1963-1986)Online publication date: 2-Apr-2024
https://dl.acm.org/doi/10.1007/s00607-024-01279-w
Chen LWang JThakuriah P(2023)Quality-Guaranteed and Cost-Effective Population Health Profiling: A Deep Active Learning ApproachACM Transactions on Computing for Healthcare10.1145/36171794:4(1-19)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3617179

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents