research-article

Towards assessing the quality of knowledge graphs via differential testing

Authors:

Yang FengAuthors Info & Claims

Volume 174, Issue C

https://doi.org/10.1016/j.infsof.2024.107521

Published: 01 October 2024 Publication History

Abstract

Knowledge graphs (KG) can aggregate data and make information resources easier to calculate and understand. With tremendous advancements in knowledge graphs, they have been incorporated into plenty of software systems to assist various tasks. However, while KGs determine the performance of downstream software systems, their quality is often measured by the accuracy of test data. Considering the limitation of accessible high-quality test data, an automated quality assessment technique could fundamentally improve the testing efficiency of KG-driven software systems and save plenty of manual labeling resources.

In this paper, we propose an automated approach to quantify the quality of KGs via differential testing. It first constructs multiple Knowledge Graph Embedding Models (KGEM) and conducts head prediction tasks on models. Then, it can produce a differential score that reflects the quality of KGs by comparing the proximity of output results. To validate the effectiveness of this approach, we experiment with four open-sourced knowledge graphs. The experiment results show that our approach is capable of accurately evaluating the quality of KGs and producing reliable results on different datasets. Moreover, we compared our method with existing methods and achieved certain advantages. The potential usefulness of our approach sheds light on the development of various KG-driven software systems.

References

[1]

Chen Y., Sinha B., Ye F., Tang T., Wu R., He M., Zheng X., Shen B., Prostate cancer management with lifestyle intervention: From knowledge graph to Chatbot, Clin. Transl. Discov. 2 (1) (2022).

[2]

Ni P., Okhrati R., Guan S., Chang V., Knowledge graph and deep learning-based text-to-GraphQL model for intelligent medical consultation chatbot, Inf. Syst. Front. 26 (1) (2024) 137–156.

[3]

Q. Bao, L. Ni, J. Liu, HHH: an online medical chatbot system based on knowledge graph and hierarchical bi-directional attention, in: Proceedings of the Australasian Computer Science Week Multiconference, 2020, pp. 1–10.

[4]

Huang S., Wang Y., Yu X., Design and implementation of oil and gas information on intelligent search engine based on knowledge graph, in: Journal of Physics: Conference Series, Vol. 1621, IOP Publishing, 2020.

[5]

Zhao X., Chen H., Xing Z., Miao C., Brain-inspired search engine assistant based on knowledge graph, IEEE Trans. Neural Netw. Learn. Syst. (2021).

[6]

Gao M., Li J.-Y., Chen C.-H., Li Y., Zhang J., Zhan Z.-H., Enhanced multi-task learning and knowledge graph-based recommender system, IEEE Trans. Knowl. Data Eng. (2023).

[7]

Y. Yang, C. Huang, L. Xia, C. Huang, Knowledge graph self-supervised rationalization for recommendation, in: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 3046–3056.

[8]

Liu J.-C., Chen C.-T., Lee C., Huang S.-H., Evolving knowledge graph representation learning with multiple attention strategies for citation recommendation system, ACM Trans. Intell. Syst. Technol. (2024).

[9]

Chen X., Jia S., Xiang Y., A review: Knowledge reasoning over knowledge graph, Expert Syst. Appl. 141 (2020).

Digital Library

[10]

Paulheim H., Knowledge graph refinement: A survey of approaches and evaluation methods, Semant. Web 8 (3) (2017) 489–508.

Digital Library

[11]

Chen Z., Wang Y., Zhao B., Cheng J., Zhao X., Duan Z., Knowledge graph completion: A review, IEEE Access 8 (2020) 192435–192456.

[12]

Hoffart J., Suchanek F.M., Berberich K., Weikum G., YAGO2: A spatially and temporally enhanced knowledge base from wikipedia, Artificial Intelligence 194 (2013) 28–61.

Digital Library

[13]

T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, J. Welling, Never-Ending Learning, in: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI-15, 2015.

[14]

Miller G.A., WordNet: An Electronic Lexical Database, MIT Press, 1998.

[15]

Bordes A., Usunier N., Garcia-Duran A., Weston J., Yakhnenko O., Translating embeddings for modeling multi-relational data, Adv. Neural Inf. Process. Syst. 26 (2013).

[16]

M. Nickel, L. Rosasco, T. Poggio, Holographic embeddings of knowledge graphs, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30, 2016.

[17]

Zhang S., Tay Y., Yao L., Liu Q., Quaternion knowledge graph embeddings, Adv. Neural Inf. Process. Syst. 32 (2019).

[18]

T. Dettmers, P. Minervini, P. Stenetorp, S. Riedel, Convolutional 2d knowledge graph embeddings, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 2018.

[19]

Wang Q., Mao Z., Wang B., Guo L., Knowledge graph embedding: A survey of approaches and applications, IEEE Trans. Knowl. Data Eng. 29 (12) (2017) 2724–2743.

[20]

Paulheim H., Bizer C., Type inference on noisy rdf data, in: International Semantic Web Conference, Springer, 2013, pp. 510–525.

[21]

H. Paulheim, J. Fümkranz, Unsupervised generation of data mining features from linked open data, in: Proceedings of the 2nd International Conference on Web Intelligence, Mining and Semantics, 2012, pp. 1–12.

[22]

M. Fabian, K. Gjergji, W. Gerhard, et al., Yago: A core of semantic knowledge unifying wordnet and wikipedia, in: 16th International World Wide Web Conference, WWW, 2007, pp. 697–706.

[23]

Chen C., Wang T., Zheng Y., Liu Y., Xie H., Deng J., Cheng L., Reinforcement learning-based distant supervision relation extraction for fault diagnosis knowledge graph construction under industry 4.0, Adv. Eng. Inform. 55 (2023).

[24]

Peifeng L., Qian L., Zhao X., Tao B., Joint knowledge graph and large language model for fault diagnosis and its application in aviation assembly, IEEE Trans. Ind. Inform. (2024).

[25]

Wienand D., Paulheim H., Detecting incorrect numerical data in dbpedia, in: European Semantic Web Conference, Springer, 2014, pp. 504–518.

[26]

Chen C., Zheng F., Cui J., Cao Y., Liu G., Wu J., Zhou J., Survey and open problems in privacy-preserving knowledge graph: merging, query, representation, completion, and applications, Int. J. Mach. Learn. Cybern. (2024) 1–20.

[27]

McKeeman W.M., Differential testing for software, Digit. Tech. J. 10 (1) (1998) 100–107.

[28]

Miller E., An introduction to the resource description framework, D-lib Mag. (1998).

[29]

Lenat D.B., CYC: A large-scale investment in knowledge infrastructure, Commun. ACM 38 (11) (1995) 33–38.

[30]

LiuQiao L., DuanHong L., et al., Knowledge graph construction techniques, J. Comput. Res Dev. 53 (3) (2016) 582.

[31]

Yang B., Yih W.-t., He X., Gao J., Deng L., Embedding entities and relations for learning and inference in knowledge bases, 2014, arXiv preprint arXiv:1412.6575.

[32]

Nickel M., Tresp V., Kriegel H.-P., A three-way model for collective learning on multi-relational data, in: ICML, 2011.

[33]

Barr E.T., Harman M., McMinn P., Shahbaz M., Yoo S., The oracle problem in software testing: A survey, IEEE Trans. Softw. Eng. 41 (5) (2014) 507–525.

Digital Library

[34]

Petsios T., Tang A., Stolfo S., Keromytis A.D., Jana S., NEZHA: Efficient domain-independent differential testing, in: 2017 IEEE Symposium on Security and Privacy, SP, 2017, pp. 615–632,.

[35]

Sotiropoulos T., Chaliasos S., Atlidakis V., Mitropoulos D., Spinellis D., Data-oriented differential testing of object-relational mapping systems, in: 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE, IEEE, 2021, pp. 1535–1547.

[36]

Gulzar M.A., Zhu Y., Han X., Perception and practices of differential testing, in: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP, 2019, pp. 71–80,.

Digital Library

[37]

Schlichtkrull A., Schou M.K., Srba J., Traytel D., Differential testing of pushdown reachability with a formally verified oracle, in: FMCAD, 2022, pp. 369–379.

[38]

Dai Y., Wang S., Xiong N.N., Guo W., A survey on knowledge graph embedding: Approaches, applications and benchmarks, Electronics 9 (5) (2020) 750.

[39]

Socher R., Chen D., Manning C.D., Ng A., Reasoning with neural tensor networks for knowledge base completion, in: Advances In Neural Information Processing Systems, 2013, pp. 926–934.

[40]

Z. Wang, J. Zhang, J. Feng, Z. Chen, Knowledge graph embedding by translating on hyperplanes, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 28, 2014.

[41]

Webber W., Moffat A., Zobel J., A similarity measure for indefinite rankings, ACM Trans. Inf. Syst. (TOIS) 28 (4) (2010) 1–38.

[42]

Ali M., Berrendorf M., Hoyt C.T., Vermue L., Sharifzadeh S., Tresp V., Lehmann J., PyKEEN 1.0: A python library for training and evaluating knowledge graph embeddings, J. Mach. Learn. Res. 22 (82) (2021) 1–6. URL http://jmlr.org/papers/v22/20-825.html.

[43]

Kemp C., Tenenbaum J.B., Griffiths T.L., Yamada T., Ueda N., Learning systems of concepts with an infinite relational model, in: AAAI, Vol. 3, 2006, p. 5.

[44]

Toutanova K., Chen D., Observed versus latent features for knowledge base and text inference, in: Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, Association for Computational Linguistics, Beijing, China, 2015, pp. 57–66,. URL https://aclanthology.org/W15-4007.

[45]

Safavi T., Koutra D., CoDEx: A comprehensive knowledge graph completion benchmark, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, Association for Computational Linguistics, Online, 2020, pp. 8328–8350,. URL https://www.aclweb.org/anthology/2020.emnlp-main.669.

[46]

Z. Cao, Q. Xu, Z. Yang, X. Cao, Q. Huang, Geometry interaction knowledge graph embeddings, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, 2022, pp. 5521–5529.

[47]

J. Yang, X. Ying, Y. Shi, X. Tong, R. Wang, T. Chen, B. Xing, Learning hierarchy-aware quaternion knowledge graph embeddings with representing relations as 3D rotations, in: Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 2011–2023.

[48]

Lehmann J., Gerber D., Morsey M., Ngomo A.-C.N., Defacto-deep fact validation, in: International Semantic Web Conference, Springer, 2012, pp. 312–327.

[49]

Waitelonis J., Ludwig N., Knuth M., Sack H., Whoknows? Evaluating linked data heuristics with a quiz that cleans up dbpedia, Interact. Technol. Smart Educ. (2011).

[50]

Siorpaes K., Hepp M., Games with a purpose for the semantic web, IEEE Intell. Syst. 23 (3) (2008) 50–60.

[51]

Fieller E.C., Hartley H.O., Pearson E.S., Tests for rank correlation coefficients. I, Biometrika 44 (3/4) (1957) 470–481.

[52]

Gao J., Li X., Xu Y.E., Sisman B., Dong X.L., Yang J., Efficient knowledge graph accuracy evaluation, 2019, arXiv preprint arXiv:1907.09657.

[53]

Y. Qi, W. Zheng, L. Hong, L. Zou, Evaluating Knowledge Graph Accuracy Powered by Optimized Human-machine Collaboration, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 1368–1378.

[54]

Akoglu H., User’s guide to correlation coefficients, Turk. J. Emerg. Med. 18 (3) (2018) 91–93.

[55]

Diaconis P., Graham R.L., Spearman’s footrule as a measure of disarray, J. R. Stat. Soc. Ser. B Stat. Methodol. 39 (2) (1977) 262–268.

[56]

Kim J., Kim E.-K., Won Y., Nam S., Choi K.-S., The association rule mining system for acquiring knowledge of dbpedia from wikipedia categories, in: NLP-DBPEDIA@ ISWC, 2015, pp. 68–80.

[57]

R. Dorsch, M. Freund, J. Fries, A. Harth, GraphGuard: Enhancing Data Quality in Knowledge Graph Pipelines, in: Proceedings of the 2nd International Workshop on Semantic Industrial Information Modelling (SemIIM 2023) Co-Located with 22nd International Semantic Web Conference, ISWC 2023, 2023.

[58]

Xue B., Zou L., Knowledge graph quality management: a comprehensive survey, IEEE Trans. Knowl. Data Eng. 35 (5) (2022) 4969–4988.

[59]

S. Marchesin, G. Silvello, Efficient and Reliable Estimation of Knowledge Graph Accuracy.

[60]

Khokhlov I., Reznik L., Knowledge graph in data quality evaluation for IoT applications, in: 2020 IEEE 6th World Forum on Internet of Things, WF-IoT, 2020, pp. 1–6,.

[61]

P. Ojha, P. Talukdar, KGEval: Accuracy estimation of automatically constructed knowledge graphs, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 1741–1750.

[62]

Zhang J.M., Harman M., Ma L., Liu Y., Machine learning testing: Survey, landscapes and horizons, IEEE Trans. Softw. Eng. (2020).

[63]

Isaku E., Laaber C., Sartaj H., Ali S., Schwitalla T., Nygård J.F., LLMs in the heart of differential testing: A case study on a medical rule engine, 2024, arXiv preprint arXiv:2404.03664.

[64]

Asyrofi M.H., Thung F., Lo D., Jiang L., CrossASR: Efficient differential testing of automatic speech recognition via text-to-speech, in: 2020 IEEE International Conference on Software Maintenance and Evolution, ICSME, 2020, pp. 640–650,.

[65]

Guo J., Jiang Y., Zhao Y., Chen Q., Sun J., DLFuzz: Differential fuzzing testing of deep learning systems, in: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, in: ESEC/FSE 2018, Association for Computing Machinery, New York, NY, USA, 2018, pp. 739–743,.

Digital Library

[66]

Zhang X., Liu J., Sun N., Fang C., Liu J., Wang J., Chai D., Chen Z., Duo: Differential fuzzing for deep learning operators, IEEE Trans. Reliab. 70 (4) (2021) 1671–1685,.

[67]

Pham H.V., Lutellier T., Qi W., Tan L., CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries, in: 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE, 2019, pp. 1027–1038,.

Digital Library

[68]

S. Li, M. Rigger, Finding XPath Bugs in XML Document Processors via Differential Testing, in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–12.

Index Terms

Towards assessing the quality of knowledge graphs via differential testing

Index terms have been assigned to the content through auto-classification.

Recommendations

Differential testing: a new approach to change detection
ESEC-FSE '07: Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering

Regression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression ...
Differential testing: a new approach to change detection
ESEC-FSE companion '07: The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering: companion papers

Regression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression ...
Entity Alignment Between Knowledge Graphs Using Entity Type Matching
Knowledge Science, Engineering and Management
Abstract
The task of entity alignment between knowledge graphs (KGs) aims to find entities in two knowledge graphs that represent the same real-world entity. Recently, embedding-based entity alignment methods get extended attention. Most of them firstly ...

Comments

Information & Contributors

Information

Published In

cover image Information and Software Technology

Information and Software Technology Volume 174, Issue C

Oct 2024

228 pages

Issue’s Table of Contents

Copyright © 2024.

Publisher

Butterworth-Heinemann

United States

Publication History

Published: 01 October 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents