A Generalization of Recurrent Neural Networks for Graph Embedding
Pages 247 - 259
Abstract
Due to the ubiquity of graphs, machine learning on graphs facilitates many AI systems. In order to incorporate the rich information of graphs into machine learning models, graph embedding has been developed, which seeks to preserve the graphs into low dimensional embeddings. Recently, researchers try to conduct graph embedding via generalizing neural networks on graphs. However, most existing approaches focus on node embedding, ignoring the heterogeneity of edges. Besides, the similarity relationship among random walk sequences has been rarely discussed. In this paper, we propose a generalization of Recurrent Neural Networks on Graphs (G-RNN) for graph embedding. More specifically, first we propose to utilize edge embedding and node embedding jointly to preserve graphs, which is of great significance in multi-relational graphs with heterogeneous edges. Then we propose the definition of subgraph level high-order proximity to preserve the inter-sequence proximity into the embeddings. To verify the generalization of G-RNN, we apply it to the embedding of knowledge graph, a typical multi-relational graph. Empirically we evaluate the resulting embeddings on the tasks of link prediction and node classification. The results show that the embeddings learned by G-RNN are powerful on both tasks, producing better performance than the baselines.
References
[1]
Ahmed, A., Shervashidze, N., Narayanamurthy, S., Josifovski, V., Smola, A.J.: Distributed large-scale natural graph factorization. In: Proceedings of the 22nd International Conference on World Wide Web, pp. 37–48. ACM (2013)
[2]
Belkin, M., Niyogi, P.: Laplacian eigenmaps and spectral techniques for embedding and clustering. In: Advances in Neural Information Processing Systems, pp. 585–591 (2002)
[3]
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1247–1250. ACM (2008)
[4]
Bordes A, Glorot X, Weston J, and Bengio Y A semantic matching energy function for learning with multi-relational data Mach. Learn. 2014 94 2 233-259
[5]
Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems, pp. 2787–2795 (2013)
[6]
Bordes, A., Weston, J., Collobert, R., Bengio, Y., et al.: Learning structured embeddings of knowledge bases. In: AAAI, vol. 6, p. 6 (2011)
[7]
Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning, pp. 160–167. ACM (2008)
[8]
Das, R., Neelakantan, A., Belanger, D., McCallum, A.: Chains of reasoning over entities, relations, and text using recurrent neural networks. arXiv preprint arXiv:1607.01426 (2016)
[9]
Garcia-Duran, A., Bordes, A., Usunier, N.: Composing relationships with translations. Ph.D. thesis, CNRS, Heudiasyc (2015)
[10]
Gardner, M., Mitchell, T.M.: Efficient and expressive knowledge base completion using subgraph feature extraction. In: EMNLP, pp. 1488–1498 (2015)
[11]
Goyal, P., Ferrara, E.: Graph embedding techniques, applications, and performance: a survey. arXiv preprint arXiv:1705.02801 (2017)
[12]
Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864. ACM (2016)
[13]
Jenatton, R., Roux, N.L., Bordes, A., Obozinski, G.R.: A latent factor model for highly multi-relational data. In: Advances in Neural Information Processing Systems, pp. 3167–3175 (2012)
[14]
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
[15]
Lao, N., Mitchell, T., Cohen, W.W.: Random walk inference and learning in a large scale knowledge base. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 529–539. Association for Computational Linguistics (2011)
[16]
Lin, Y., Liu, Z., Luan, H., Sun, M., Rao, S., Liu, S.: Modeling relation paths for representation learning of knowledge bases. arXiv preprint arXiv:1506.00379 (2015)
[17]
Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embeddings for knowledge graph completion. In: AAAI, pp. 2181–2187 (2015)
[18]
Neelakantan, A., Chang, M.W.: Inferring missing entity type instances for knowledge base completion: new dataset and methods. arXiv preprint arXiv:1504.06658 (2015)
[19]
Neelakantan, A., Roth, B., McCallum, A.: Compositional vector space models for knowledge base inference. In: 2015 AAAI Spring Symposium Series (2015)
[20]
Nickel, M., Tresp, V., Kriegel, H.P.: A three-way model for collective learning on multi-relational data. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 809–816 (2011)
[21]
Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 701–710. ACM (2014)
[22]
Ristoski P and Paulheim H Groth P, Simperl E, Gray A, Sabou M, Krötzsch M, Lecue F, Flöck F, and Gil Y RDF2Vec: RDF graph embeddings for data mining The Semantic Web – ISWC 2016 2016 Cham Springer 498-514
[23]
Roweis ST and Saul LK Nonlinear dimensionality reduction by locally linear embedding Science 2000 290 5500 2323-2326
[24]
Schlichtkrull, M., Kipf, T.N., Bloem, P., Berg, R.v.d., Titov, I., Welling, M.: Modeling relational data with graph convolutional networks. arXiv preprint arXiv:1703.06103 (2017)
[25]
Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., Mei, Q.: Line: large-scale information network embedding. In: Proceedings of the 24th International Conference on World Wide Web, pp. 1067–1077. International World Wide Web Conferences Steering Committee (2015)
[26]
Wang, D., Cui, P., Zhu, W.: Structural deep network embedding. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1225–1234. ACM (2016)
[27]
Wang, Z., Zhang, J., Feng, J., Chen, Z.: Knowledge graph embedding by translating on hyperplanes. In: AAAI, pp. 1112–1119 (2014)
[28]
Xie, R., Liu, Z., Chua, T.s., Luan, H., Sun, M.: Image-embodied knowledge representation learning. arXiv preprint arXiv:1609.07028 (2016)
[29]
Xie, R., Liu, Z., Jia, J., Luan, H., Sun, M.: Representation learning of knowledge graphs with entity descriptions. In: AAAI, pp. 2659–2665 (2016)
Recommendations
Embedding Watermarks into Deep Neural Networks
ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia RetrievalSignificant progress has been made with deep neural networks recently. Sharing trained models of deep neural networks has been a very important in the rapid progress of research and development of these systems. At the same time, it is necessary to ...
Node embedding from neural Hamiltonian orbits in graph neural networks
ICML'23: Proceedings of the 40th International Conference on Machine LearningIn the graph node embedding problem, embedding spaces can vary significantly for different data types, leading to the need for different GNN model types. In this paper, we model the embedding update of a node feature as a Hamiltonian orbit over time. ...
Comments
Information & Contributors
Information
Published In
Jun 2018
621 pages
ISBN:978-3-319-93036-7
DOI:10.1007/978-3-319-93037-4
© Springer International Publishing AG, part of Springer Nature 2018.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 20 June 2018
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 20 Jan 2025