Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

A Normalizing Flow-Based Co-Embedding Model for Attributed Networks

Published: 22 October 2021 Publication History

Abstract

Network embedding is a technique that aims at inferring the low-dimensional representations of nodes in a semantic space. In this article, we study the problem of inferring the low-dimensional representations of both nodes and attributes for attributed networks in the same semantic space such that the affinity between a node and an attribute can be effectively measured. Intuitively, this problem can be addressed by simply utilizing existing variational auto-encoder (VAE) based network embedding algorithms. However, the variational posterior distribution in previous VAE based network embedding algorithms is often assumed and restricted to be a mean-field Gaussian distribution or other simple distribution families, which results in poor inference of the embeddings. To alleviate the above defect, we propose a novel VAE-based co-embedding method for attributed network, F-CAN, where posterior distributions are flexible, complex, and scalable distributions constructed through the normalizing flow. We evaluate our proposed models on a number of network tasks with several benchmark datasets. Experimental results demonstrate that there are clear improvements in the qualities of embeddings generated by our model to the state-of-the-art attributed network embedding methods.

References

[1]
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. 265–283.
[2]
Lada A. Adamic and Eytan Adar. 2003. Friends and neighbors on the web. Social Networks 25, 3 (2003), 211–230.
[3]
Amr Ahmed, Nino Shervashidze, Shravan M. Narayanamurthy, Vanja Josifovski, and Alexander J. Smola. 2013. Distributed large-scale natural graph factorization. In Proceedings of the 22nd International World Wide Web Conference. 37–48.
[4]
Benjamin Aubin, Bruno Loureiro, Antoine Maillard, Florent Krzakala, and Lenka Zdeborová. 2021. The spiked matrix model with generative priors. IEEE Transactions on Information Theory 67, 2 (2021), 1156–1181.
[5]
Mislav Balunović, Anian Ruoss, and Martin Vechev. 2021. Fair normalizing flows. arXiv:2106.05937. Retrieved from https://arxiv.org/abs/2106.05937.
[6]
Aleksandar Bojchevski and Stephan Günnemann. 2018. Deep Gaussian embedding of graphs: Unsupervised inductive learning via ranking. In Proceedings of the 6th International Conference on Learning Representations.
[7]
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In Proceedings of the 4th International Conference on Learning Representations.
[8]
Shaosheng Cao, Wei Lu, and Qiongkai Xu. 2015. GraRep: Learning graph representations with global structural information. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. 891–900.
[9]
Deepayan Chakrabarti, Stanislav Funiak, Jonathan Chang, and Sofus A. Macskassy. 2014. Joint inference of multiple label types in large networks. In Proceedings of the 31st International Conference on Machine Learning. 874–882.
[10]
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. 2018. Hyperspherical variational auto-encoders. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence. 856–865.
[11]
Laurent Dinh, David Krueger, and Yoshua Bengio. 2015. NICE: Non-linear independent components estimation. In Proceedings of the 3rd International Conference on Learning Representations.
[12]
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2017. Density estimation using real NVP. In Proceedings of the 5th International Conference on Learning Representations.
[13]
Marylou Gabrié. 2020. Mean-field inference methods for neural networks. Journal of Physics A: Mathematical and Theoretical 53, 22 (2020), 223002.
[14]
Marylou Gabrié, Andre Manoel, Clément Luneau, Jean Barbier, Nicolas Macris, Florent Krzakala, and Lenka Zdeborová. 2018. Entropy and mutual information in models of deep neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 1826–1836.
[15]
Christina Gao, Stefan Höche, Joshua Isaacson, Claudius Krause, and Holger Schulz. 2020. Event generation with normalizing flows. Physical Review D 101, 7 (2020), 076002.
[16]
Christina Gao, Joshua Isaacson, and Claudius Krause. 2020. i-flow: High-dimensional integration and sampling with normalizing flows. Machine Learning: Science and Technology 1, 4 (2020), 045023.
[17]
Hongchang Gao and Heng Huang. 2018. Deep attributed network embedding. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 3364–3370.
[18]
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. 2015. MADE: Masked autoencoder for distribution estimation. In Proceedings of the 32nd International Conference on Machine Learning. 881–889.
[19]
Neil Zhenqiang Gong, Ameet Talwalkar, Lester Mackey, Ling Huang, Eui Chul Richard Shin, Emil Stefanov, Elaine Shi, and Dawn Song. 2014. Joint link prediction and attribute inference using a social-attribute network. ACM Transactions on Intelligent Systems and Technology 5, 2 (2014), 1–20.
[20]
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning. 1462–1471.
[21]
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 855–864.
[22]
William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 1025–1035.
[23]
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of the 5th International Conference on Learning Representations.
[24]
Xiao Huang, Jundong Li, and Xia Hu. 2017. Accelerated attributed network embedding. In Proceedings of the 2017 SIAM International Conference on Data Mining. 633–641.
[25]
Xiao Huang, Jundong Li, and Xia Hu. 2017. Label informed attributed network embedding. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining. 731–739.
[26]
Xiao Huang, Qingquan Song, Jundong Li, and Xia Hu. 2018. Exploring expert cognition for attributed network embedding. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 270–278.
[27]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.
[28]
Diederik P. Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1 \(\times\) 1 convolutions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 10215–10224.
[29]
Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Proceedings of the 27th International Conference on Neural Information Processing Systems. 3581–3589.
[30]
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improving variational inference with inverse autoregressive flow. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 1–10.
[31]
Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations.
[32]
Thomas N. Kipf and Max Welling. 2016. Variational graph auto-encoders. stat 1050 (2016), 21 pages.
[33]
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations.
[34]
Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning. 1188–1196.
[35]
Jundong Li, Kewei Cheng, Liang Wu, and Huan Liu. 2018. Streaming link prediction on dynamic attributed networks. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 369–377.
[36]
Jundong Li, Harsh Dani, Xia Hu, Jiliang Tang, Yi Chang, and Huan Liu. 2017. Attributed network embedding for learning in a dynamic environment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 387–396.
[37]
Ye Li, Chaofeng Sha, Xin Huang, and Yanchun Zhang. 2018. Community detection in attributed graphs: An embedding approach. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 338–345.
[38]
Jiongqian Liang, Peter Jacobs, Jiankai Sun, and Srinivasan Parthasarathy. 2018. Semi-supervised embedding in attributed networks with outliers. In Proceedings of the 2018 SIAM International Conference on Data Mining. 153–161.
[39]
Shangsong Liang. 2018. Dynamic user profiling for streams of short texts. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32.
[40]
Shangsong Liang. 2019. Collaborative, dynamic and diversified user profiling. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 4269–4276.
[41]
Shangsong Liang, Emine Yilmaz, and Evangelos Kanoulas. 2018. Collaboratively tracking interests for user clustering in streams of short texts. IEEE Transactions on Knowledge and Data Engineering 31, 2 (2018), 257–272.
[42]
Shangsong Liang, Xiangliang Zhang, Zhaochun Ren, and Evangelos Kanoulas. 2018. Dynamic embeddings for user profiling in Twitter. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1764–1773.
[43]
Siyuan Liao, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. 2021. Learning dynamic embeddings for temporal knowledge graphs. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 535–543.
[44]
Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. 2019. Graph normalizing flows. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. 13556–13566.
[45]
Zhining Liu, Dawei Zhou, and Jingrui He. 2019. Towards explainable representation of time-evolving graphs via spatial-temporal graph attention networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2137–2140.
[46]
Zhining Liu, Dawei Zhou, Yada Zhu, Jinjie Gu, and Jingrui He. 2020. Towards fine-grained temporal network representation via time-reinforced random walk. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, 4973–4980.
[47]
Yupeng Luo, Shangsong Liang, and Zaiqiao Meng. 2019. Constrained co-embedding model for user profiling in question answering communities. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 439–448.
[48]
Julian J. McAuley and Jure Leskovec. 2012. Learning to discover social circles in ego networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems.548–556.
[49]
Zaiqiao Meng, Shangsong Liang, Hongyan Bao, and Xiangliang Zhang. 2019. Co-embedding attributed networks. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining. 393–401.
[50]
Zaiqiao Meng, Shangsong Liang, Jinyuan Fang, and Teng Xiao. 2019. Semi-supervisedly co-embedding attributed networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. 4743–4751.
[51]
Zaiqiao Meng, Shangsong Liang, Xiangliang Zhang, Richard McCreadie, and Iadh Ounis. 2020. Jointly learning representations of nodes and attributes for attributed networks. ACM Transactions on Information Systems 38, 2 (2020), 1–32.
[52]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Advances in Neural Information processing Systems. 3111–3119.
[53]
Andriy Mnih and Karol Gregor. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on International Conference on Machine Learning. 1791–1799.
[54]
Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu. 2016. Asymmetric transitivity preserving graph embedding. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1105–1114.
[55]
Gaurav Pandey and Ambedkar Dukkipati. 2016. Variational methods for conditional multimodal learning: Generating human faces from attributes. arXiv:1603.01801. Retrieved from https://arxiv.org/abs/1603.01801.
[56]
George Papamakarios, Iain Murray, and Theo Pavlakou. 2017. Masked autoregressive flow for density estimation. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 2338–2347.
[57]
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 701–710.
[58]
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. 2016. Variational autoencoder for deep learning of images, labels and captions. In Proceedings of the 30th International Conference on Neural Information Processing Systems. Vol. 29, 2352–2360.
[59]
Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference on Machine Learning. PMLR, 1530–1538.
[60]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on International Conference on Machine Learning. 1278–1286.
[61]
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI Magazine 29, 3 (2008), 93.
[62]
Martin Simonovsky and Nikos Komodakis. 2018. GraphVAE: Towards generation of small graphs using variational autoencoders. In Proceedings of the International Conference on Artificial Neural Networks. Springer, 412–422.
[63]
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. Learning structured output representation using deep conditional generative models. In Proceedings of the 28th International Conference on Neural Information Processing Systems. 3483–3491.
[64]
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. 2016. Ladder variational autoencoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 3738–3746.
[65]
Heli Sun, Fang He, Jianbin Huang, Yizhou Sun, Yang Li, Chenyu Wang, Liang He, Zhongbin Sun, and Xiaolin Jia. 2020. Network embedding for community detection in attributed networks. ACM Transactions on Knowledge Discovery from Data 14, 3 (2020), 1–25.
[66]
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015. LINE: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web. 1067–1077.
[67]
Lei Tang and Huan Liu. 2009. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 817–826.
[68]
Richard Turner, M. Sahani. 2011. Two problems with variational expectation maximisation for time-series models. In Bayesian Time Series Models. D. Barber, T. Cemgil, and S. Chiappa (Eds.), Cambridge University Press, 109–130.
[69]
Hao Wang, Enhong Chen, Qi Liu, Tong Xu, Dongfang Du, Wen Su, and Xiaopeng Zhang. 2018. A united approach to learning sparse attributed network embedding. In Proceedings of the 2018 IEEE International Conference on Data Mining. 557–566.
[70]
Suhang Wang, Charu Aggarwal, Jiliang Tang, and Huan Liu. 2017. Attributed signed network embedding. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 137–146.
[71]
Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, and Shiqiang Yang. 2017. Community preserving network embedding. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.
[72]
Hao Wu, Jonas Köhler, and Frank Noé. 2020. Stochastic normalizing flows. arXiv:2002.06707. Retrieved from https://arxiv.org/abs/2002.06707.
[73]
Wei Wu, Bin Li, Ling Chen, and Chengqi Zhang. 2018. Efficient attributed network embedding via recursive randomized hashing. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2861–2867.
[74]
Carl Yang, Lin Zhong, Li-Jia Li, and Luo Jie. 2017. Bi-directional joint inference for user links and attributes on large social graphs. In Proceedings of the 26th International Conference on World Wide Web. 564–573.
[75]
Hong Yang, Shirui Pan, Peng Zhang, Ling Chen, Defu Lian, and Chengqi Zhang. 2018. Binarized attributed network embedding. In Proceedings of the 2018 IEEE International Conference on Data Mining. 1476–1481.
[76]
Andrei Zanfir, Eduard Gabriel Bazavan, Hongyi Xu, Bill Freeman, Rahul Sukthankar, and Cristian Sminchisescu. 2020. Weakly supervised 3d human pose and shape reconstruction with normalizing flows. In Proceedings of the European Conference on Computer Vision. Springer, 465–481.
[77]
Chengxi Zang and Fei Wang. 2020. MoFlow: An invertible flow model for generating molecular graphs. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 617–626.
[78]
Zhen Zhang, Hongxia Yang, Jiajun Bu, Sheng Zhou, Pinggang Yu, Jianwei Zhang, Martin Ester, and Can Wang. 2018. ANRL: Attributed network representation learning via deep neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 3155–3161.
[79]
Dawei Zhou, Jingrui He, Hongxia Yang, and Wei Fan. 2018. Sparc: Self-paced network representation for few-shot rare category characterization. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2807–2816.
[80]
Dingyuan Zhu, Peng Cui, Daixin Wang, and Wenwu Zhu. 2018. Deep variational network embedding in Wasserstein space. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2827–2836.

Cited By

View all
  • (2024)Active Learning for Multi-Class Drug-Drug Interactions Prediction2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)10.1109/BIBM62325.2024.10822197(1611-1615)Online publication date: 3-Dec-2024
  • (2023)Attention‐based network embedding with higher‐order weights and node attributesCAAI Transactions on Intelligence Technology10.1049/cit2.12215Online publication date: 5-Apr-2023
  • (2022)Cross-Temporal Snapshot Alignment for Dynamic Multi-Relational NetworksJournal of Physics: Conference Series10.1088/1742-6596/2253/1/0120382253:1(012038)Online publication date: 1-Apr-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data
ACM Transactions on Knowledge Discovery from Data  Volume 16, Issue 3
June 2022
494 pages
ISSN:1556-4681
EISSN:1556-472X
DOI:10.1145/3485152
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2021
Accepted: 01 July 2021
Revised: 01 February 2021
Received: 01 May 2020
Published in TKDD Volume 16, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Attributed network
  2. embedding
  3. variational auto-encoder
  4. normalizing flow

Qualifiers

  • Research-article
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)45
  • Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Active Learning for Multi-Class Drug-Drug Interactions Prediction2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)10.1109/BIBM62325.2024.10822197(1611-1615)Online publication date: 3-Dec-2024
  • (2023)Attention‐based network embedding with higher‐order weights and node attributesCAAI Transactions on Intelligence Technology10.1049/cit2.12215Online publication date: 5-Apr-2023
  • (2022)Cross-Temporal Snapshot Alignment for Dynamic Multi-Relational NetworksJournal of Physics: Conference Series10.1088/1742-6596/2253/1/0120382253:1(012038)Online publication date: 1-Apr-2022
  • (2022)Multi-granular attributed network representation learningInternational Journal of Machine Learning and Cybernetics10.1007/s13042-022-01507-913:7(2071-2087)Online publication date: 2-Mar-2022

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media