ReliK: A Reliability Measure for Knowledge Graph Embeddings
Authors: 
Maximilian K. Egger, Wenyue Ma, Davide Mottin, Panagiotis Karras, Ilaria Bordino, 
Francesco Gullo, Aris AnagnostopoulosAuthors Info & Claims
Maximilian K. Egger, Wenyue Ma, Davide Mottin, Panagiotis Karras, Ilaria Bordino, Francesco Gullo, Aris AnagnostopoulosAuthors Info & ClaimsPages 2009 - 2019
Abstract
Can we assess a priori how well a knowledge graph embedding will perform on a specific downstream task and in a specific part of the knowledge graph? Knowledge graph embeddings (KGEs) represent entities (e.g., ''da Vinci,'' ''Mona Lisa'') and relationships (e.g., ''painted'') of a knowledge graph (KG) as vectors. KGEs are generated by optimizing an embedding score, which assesses whether a triple (e.g., ''da Vinci,'' "painted,'' ''Mona Lisa'') exists in the graph. KGEs have been proven effective in a variety of web-related downstream tasks, including, for instance, predicting relationship(s) among entities. However, the problem of anticipating the performance of a given KGE in a certain downstream task and locally to a specific individual triple, has not been tackled so far.
In this paper, we fill this gap withReliK, a Reli ability measure for K GEs. ReliK relies solely on KGE embedding scores, is task- and KGE-agnostic, and requires no further KGE training. As such, it is particularly appealing for semantic web applications which call for testing multiple KGE methods on various parts of the KG and on each individual downstream task. Through extensive experiments, we attest thatReliK correlates well with both common downstream tasks, such as tail/relation prediction and triple classification, as well as advanced downstream tasks, such as rule mining and question answering, while preserving locality.
Supplemental Material
MP4 File
Supplemental video
- Download
- 25.89 MB
References
[1]
Rakesh Agrawal, Ramakrishnan Srikant, et al. 1994. Fast algorithms for mining association rules. In VLDB, Vol. 1215. Santiago, Chile, 487--499.
[2]
Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Mikhail Galkin, Sahand Sharifzadeh, Asja Fischer, Volker Tresp, and Jens Lehmann. 2021. Bringing light into the dark: A large-scale evaluation of knowledge graph embedding models under a unified framework. TPAMI, Vol. 44, 12 (2021), 8825--8845.
[3]
Yushi Bai, Xin Lv, Juanzi Li, and Lei Hou. 2023. Answering Complex Logical Queries on Knowledge Graphs via Query Computation Tree Optimization. (2023).
[4]
Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. TuckER: Tensor Factorization for Knowledge Graph Completion. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 5184--5193.
[5]
Satadisha Saha Bhowmick, Eduard C. Dragut, and Weiyi Meng. 2023. Globally Aware Contextual Embeddings for Named Entity Recognition in Social Media Streams. In ICDE. 1544--1557.
[6]
Federico Bianchi, Gaetano Rossiello, Luca Costabello, Matteo Palmonari, and Pasquale Minervini. 2020. Knowledge Graph Embeddings and Explainable AI. In Knowledge Graphs for eXplainable Artificial Intelligence: Foundations, Applications and Challenges. IOS Press, 49--72.
[7]
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD. 1247--1250.
[8]
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. NeurIPS, Vol. 26 (2013).
[9]
Guillaume Bouchard, Sameer Singh, and Theo Trouillon. 2015. On approximate reasoning capabilities of low-rank vector spaces. In AAAI.
[10]
Linlin Chao, Jianshan He, Taifeng Wang, and Wei Chu. 2021. PairRE: Knowledge Graph Embeddings via Paired Relation Vectors. In ACL. 4360--4369.
[11]
Moses Charikar. 2003. Greedy approximation algorithms for finding dense components in a graph. In APPROX. Springer, 84--95.
[12]
Nick Craswell. 2009. Mean Reciprocal Rank. Encyclopedia of database systems, Vol. 1703 (2009).
[13]
Caglar Demir, Julian Lienen, and Axel-Cyrille Ngonga Ngomo. 2022. Kronecker decomposition for knowledge graph embeddings. In HT. 1--10.
[14]
Omkar Deshpande, Digvijay S Lamba, Michel Tourn, Sanjib Das, Sri Subramaniam, Anand Rajaraman, Venky Harinarayan, and AnHai Doan. 2013. Building, maintaining, and using knowledge bases: a report from the trenches. In SIGMOD. 1209--1220.
[15]
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In AAAI, Vol. 32.
[16]
Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. 2015. Fast rule mining in ontological knowledge bases with AMIE. VLDBJ, Vol. 24, 6 (2015), 707--730.
[17]
Luis Antonio Galá rraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. 2013. AMIE: association rule mining under incomplete evidence in ontological knowledge bases. In TheWebConf. 413--422.
[18]
Wassily Hoeffding. 1963. Probability Inequalities for Sums of Bounded Random Variables. J. Amer. Statist. Assoc., Vol. 58, 301 (1963), 13--30.
[19]
Johannes Hoffart, Fabian M Suchanek, Klaus Berberich, and Gerhard Weikum. 2013. YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia. Artificial intelligence, Vol. 194 (2013), 28--61.
[20]
Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d'Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan F. Sequeda, Steffen Staab, and Antoine Zimmermann. 2022. Knowledge Graphs. ACM CSUR, Vol. 54, 4 (2022), 71:1--71:37.
[21]
Jiacheng Huang, Zequn Sun, Qijin Chen, Xiaozhou Xu, Weijun Ren, and Wei Hu. 2023. Deep Active Alignment of Knowledge Graph Entities and Schemata. PACMMOD, Vol. 1, 2 (2023), 159:1--159:26.
[22]
Nitisha Jain, Jan-Christoph Kalo, Wolf-Tilo Balke, and Ralf Krestel. 2021. Do Embeddings Actually Capture Knowledge Graph Semantics?. In ESWC. 143--159.
[23]
Johan Ludwig William Valdemar Jensen. 1906. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta mathematica, Vol. 30, 1 (1906), 175--193.
[24]
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. 2021. A survey on knowledge graphs: Representation, acquisition, and applications. Trans. Neural Netw. Learn. Syst., Vol. 33, 2 (2021), 494--514.
[25]
Matteo Lissandrini, Davide Mottin, Themis Palpanas, Dimitra Papadimitriou, and Yannis Velegrakis. 2015. Unleashing the power of information graphs. ACM SIGMOD Record, Vol. 43, 4 (2015), 21--26.
[26]
Michael Loster, Davide Mottin, Paolo Papotti, Jan Ehmüller, Benjamin Feldmann, and Felix Naumann. 2021. Few-shot knowledge validation using rules. In TheWebConf. 3314--3324.
[27]
Maximilian Nickel, Volker Tresp, Hans-Peter Kriegel, et al. 2011. A three-way model for collective learning on multi-relational data. In ICML, Vol. 11. 3104482--3104584.
[28]
Stefano Ortona, Venkata Vamsikrishna Meduri, and Paolo Papotti. 2018. Robust discovery of positive and negative rules in knowledge bases. In ICDE. 1168--1179.
[29]
Stavros Papadopoulos, Lixing Wang, Yin Yang, Dimitris Papadias, and Panagiotis Karras. 2011. Authenticated Multistep Nearest Neighbor Search. IEEE Trans. Knowl. Data Eng., Vol. 23, 5 (2011), 641--654.
[30]
Heiko Paulheim. 2018. Make embeddings semantic again!. In ISWC.
[31]
Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances in Neural Information Processing Systems, Vol. 33 (2020), 19716--19726.
[32]
Tara Safavi and Danai Koutra. 2020. CoDEx: A Comprehensive Knowledge Graph Completion Benchmark. In EMNLP. 8328--8350.
[33]
Tara Safavi, Danai Koutra, and Edgar Meij. 2020. Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction. In EMNLP.
[34]
Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip. 2016. A survey of heterogeneous information network analysis. TKDE, Vol. 29, 1 (2016), 17--37.
[35]
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In TheWebConf. 697--706.
[36]
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In ICLR.
[37]
Pedro Tabacof and Luca Costabello. 2020. Probability Calibration for Knowledge Graph Embedding Models. In ICLR.
[38]
Sudhanshu Tiwari, Iti Bansal, and Carlos R Rivero. 2021. Revisiting the evaluation protocol of knowledge graph completion methods for link prediction. In TheWebConf. 809--820.
[39]
Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan. 2006. Fast random walk with restart and its applications. In ICDM. IEEE, 613--622.
[40]
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality. 57--66.
[41]
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML. PMLR, 2071--2080.
[42]
Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Ivan V. Oseledets, and Emmanuel Mü ller. 2021. FREDE: Anytime Graph Embeddings. Proc. VLDB Endow., Vol. 14, 6 (2021), 1102--1110.
[43]
Ledyard R Tucker. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika, Vol. 31, 3 (1966), 279--311.
[44]
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2019. Composition-based Multi-Relational Graph Convolutional Networks. In International Conference on Learning Representations.
[45]
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. TKDE, Vol. 29, 12 (2017), 2724--2743.
[46]
Xiangyu Wang, Lyuzhou Chen, Taiyu Ban, Muhammad Usman, Yifeng Guan, Shikang Liu, Tianhao Wu, and Huanhuan Chen. 2021. Knowledge graph quality control: A survey. Fundamental Research, Vol. 1, 5 (2021), 607--626.
[47]
Gerhard Weikum. 2021. Knowledge Graphs 2021: A Data Odyssey. PVLDB, Vol. 14, 12 (2021), 3233--3238.
[48]
Yuhan Wu, Yuanyuan Xu, Xuemin Lin, and Wenjie Zhang. 2023. A Holistic Approach for Answering Logical Queries on Knowledge Graphs. In ICDE. 2345--2357.
[49]
Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In ICLR.
[50]
Mohamad Zamini, Hassan Reza, and Minou Rabiei. 2022. A Review of Knowledge Graph Completion. Information, Vol. 13, 8 (2022), 396.
[51]
Alexandros Zeakis, George Papadakis, Dimitrios Skoutas, and Manolis Koubarakis. 2023. Pre-trained Embeddings for Entity Resolution: An Experimental Analysis. PVLDB, Vol. 16, 9 (2023), 2225--2238.
[52]
Ziyue Zhong, Meihui Zhang, Ju Fan, and Chenxiao Dou. 2022. Semantics Driven Embedding Learning for Effective Entity Alignment. In ICDE. 2127--2140.
Index Terms
- ReliK: A Reliability Measure for Knowledge Graph Embeddings
Recommendations
A survey on knowledge graph embeddings with literals: Which model links better literal-ly?
Advancing Agriculture through Semantic Data ManagementKnowledge Graphs (KGs) are composed of structured information about a particular domain in the form of entities and relations. In addition to the structured information KGs help in facilitating interconnectivity and interoperability between different ...
Knowledge graph embedding methods for entity alignment: experimental review
AbstractIn recent years, we have witnessed the proliferation of knowledge graphs (KG) in various domains, aiming to support applications like question answering, recommendations, etc. A frequent task when integrating knowledge from different KGs is to ...
Do Embeddings Actually Capture Knowledge Graph Semantics?
The Semantic WebAbstractKnowledge graph embeddings that generate vector space representations of knowledge graph triples, have gained considerable popularity in past years. Several embedding models have been proposed that achieve state-of-the-art performance for the task ...
Comments
Information & Contributors
Information
Published In
May 2024
4826 pages
ISBN:9798400701719
DOI:10.1145/3589334
- General Chairs:
- Tat-Seng Chua,
- Chong-Wah Ngo,
- Proceedings Chair:
- Roy Ka-Wei Lee,
- Program Chairs:
- Ravi Kumar,
- Hady W. Lauw
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 May 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- Horizon Europe and Innovation Fund Denmark
- PNRR MUR
- MUR PRIN
- Danish Council for Independent Research
- China Scholarship Council
- EC H2020RIA
- European Commission - NextGenerationEU
- ERC Advanced Grant
Conference
WWW '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 387Total Downloads
- Downloads (Last 12 months)387
- Downloads (Last 6 weeks)34
Reflects downloads up to 31 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in