research-article

Swift and Sure: Hardness-aware Contrastive Learning for Low-dimensional Knowledge Graph Embeddings

Authors:

Quan Z. ShengAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 838 - 849

https://doi.org/10.1145/3485447.3511927

Published: 25 April 2022 Publication History

Abstract

Knowledge graph embedding (KGE) has shown great potential in automatic knowledge graph (KG) completion and knowledge-driven tasks. However, recent KGE models suffer from high training cost and large storage space, thus limiting their practicality in real-world applications. To address this challenge, based on the latest findings in the field of Contrastive Learning, we propose a novel KGE training framework called Hardness-aware Low-dimensional Embedding (HaLE). Instead of the traditional Negative Sampling, we design a new loss function based on query sampling that can balance two important training targets, Alignment and Uniformity. Furthermore, we analyze the hardness-aware ability of recent low-dimensional hyperbolic models and propose a lightweight hardness-aware activation mechanism, which can help the KGE models focus on hard instances and speed up convergence. The experimental results show that in the limited training time, HaLE can effectively improve the performance and training speed of KGE models on five commonly-used datasets. After training just a few minutes, the HaLE-trained models are competitive compared to the state-of-the-art models in both low- and high-dimensional conditions.

References

[1]

Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. TuckER: Tensor Factorization for Knowledge Graph Completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China, 5185–5194.

[2]

Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019. Multi-relational Poincaré Graph Embeddings. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. 4465–4475.

[3]

Antoine Bordes, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems, December 5-8, 2013, Lake Tahoe, Nevada, United States. 2787–2795.

[4]

Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2014. A Semantic Matching Energy Function for Learning with Multi-relational Data. Machine Learning 94(2014), 233–259.

Digital Library

[5]

Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. 2020. Low-Dimensional Hyperbolic Knowledge Graph Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. 6901–6914.

[6]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. 1597–1607.

[7]

Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D Knowledge Graph Embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. 1811–1818.

[8]

Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge Graph Embedding via Dynamic Mapping Matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Beijing, China, 687–696.

[9]

Zelong Li, Jianchao Ji, Zuohui Fu, Yingqiang Ge, Shuyuan Xu, Chong Chen, and Yongfeng Zhang. 2021. Efficient Non-Sampling Knowledge Graph Embedding. In Proceedings of WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. 1727–1736.

Digital Library

[10]

Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In Proceedings of the 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.

[11]

Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2018. A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). New Orleans, Louisiana, 327–333.

[12]

Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A Three-Way Model for Collective Learning on Multi-Relational Data. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011. 809–816.

Digital Library

[13]

Xutan Peng, Guanyi Chen, Chenghua Lin, and Mark Stevenson. 2021. Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. 2364–2375.

[14]

Paolo Rosso, Dingqi Yang, Natalia Ostapuk, and Philippe Cudré-Mauroux. 2021. RETA: A Schema-Aware, End-to-End Solution for Instance Completion in Knowledge Graphs. In Proceedings of the WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. 845–856.

Digital Library

[15]

Daniel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. 2020. You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.

[16]

Mrinmaya Sachan. 2020. Knowledge Graph Embedding Compression. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. 2681–2691.

[17]

Tara Safavi and Danai Koutra. 2020. CoDEx: A Comprehensive Knowledge Graph Completion Benchmark. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. 8328–8350.

[18]

Tara Safavi, Danai Koutra, and Edgar Meij. 2020. Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. 8308–8321.

[19]

Graciela S.Birman and Abraham A.Ungar. 2001. The hyperbolic derivative in the Poincaré ball model of hyperbolic geometry. J. Math. Anal. Appl. 254(2001), 321–333.

[20]

Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling Relational Data with Graph Convolutional Networks. In Proceedings of the 15th Extended Semantic Web Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018. 593–607.

Digital Library

[21]

Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-End Structure-Aware Convolutional Networks for Knowledge Base Completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. AAAI Press, 3060–3067.

Digital Library

[22]

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.

[23]

Pedro Tabacof and Luca Costabello. 2020. Probability Calibration for Knowledge Graph Embedding Models. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.

[24]

Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. 57–66.

[25]

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex Embeddings for Simple Link Prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016. 2071–2080.

Digital Library

[26]

Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha P. Talukdar. 2020. Composition-based Multi-Relational Graph Convolutional Networks. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.

[27]

Feng Wang and Huaping Liu. 2021. Understanding the Behaviour of Contrastive Loss. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. 2495–2504.

[28]

Kai Wang, Yu Liu, Dan Lin, and Michael Sheng. 2021. Hyperbolic Geometry is Not Necessary: Lightweight Euclidean-Based Models for Low-Dimensional Knowledge Graph Embeddings. In Findings of EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021. 464–474.

[29]

Kai Wang, Yu Liu, Qian Ma, and Quan Z. Sheng. 2021. MulDE: Multi-teacher Knowledge Distillation for Low-dimensional Knowledge Graph Embeddings. In Proceedings of the WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. 1716–1726.

Digital Library

[30]

Menghan Wang, Mingming Gong, Xiaolin Zheng, and Kun Zhang. 2018. Modeling Dynamic Missingness of Implicit Feedback for Recommendation. In Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. 6670–6679.

[31]

Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge Graph Embedding: A Survey of Approaches and Applications. IEEE Transactions on Knowledge and Data Engineering 29, 12(2017), 2724–2743.

[32]

Shen Wang, Xiaokai Wei, Cicero Nogueira dos Santos, Zhiguo Wang, Ramesh Nallapati, Andrew O. Arnold, Bing Xiang, Philip S. Yu, and Isabel F. Cruz. 2021. Mixed-Curvature Multi-Relational Graph Neural Network for Knowledge Graph Completion. In Proceedings of the WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. 1761–1771.

Digital Library

[33]

Tongzhou Wang and Phillip Isola. 2020. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. 9929–9939.

[34]

Xiang Wang, Tinglin Huang, Dingxian Wang, Yancheng Yuan, Zhenguang Liu, Xiangnan He, and Tat-Seng Chua. 2021. Learning Intents behind Interactions with Knowledge Graph for Recommendation. In Proceedings of the WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. 878–887.

Digital Library

[35]

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015.

[36]

Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. Quaternion Knowledge Graph Embeddings. In Proceedings of the Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. 2731–2741.

[37]

Zhanqiu Zhang, Jianyu Cai, and Jie Wang. 2020. Duality-Induced Regularizer for Tensor Factorization Based Knowledge Graph Completion. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

[38]

Yushan Zhu, Wen Zhang, Hui Chen, Xu Cheng, Wei Zhang, and Huajun Chen. 2020. DistilE: Distiling Knowledge Graph Embeddings for Faster and Cheaper Reasoning. CoRR abs/2009.05912(2020). https://arxiv.org/abs/2009.05912

Cited By

Zhang HZhang JMolybog IChua TNgo CKa-Wei Lee RKumar RLauw H(2024)HaSa: Hardness and Structure-Aware Contrastive Knowledge Graph EmbeddingProceedings of the ACM on Web Conference 202410.1145/3589334.3645564(2116-2127)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645564
Jiang LYan GLuo HChang W(2023)Improved Collaborative Recommendation Model: Integrating Knowledge Embedding and Graph Contrastive LearningElectronics10.3390/electronics1220423812:20(4238)Online publication date: 13-Oct-2023
https://doi.org/10.3390/electronics12204238
Zhong LWu JLi QPeng HWu X(2023)A Comprehensive Survey on Automatic Knowledge Graph ConstructionACM Computing Surveys10.1145/361829556:4(1-62)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3618295
Show More Cited By

Index Terms

Swift and Sure: Hardness-aware Contrastive Learning for Low-dimensional Knowledge Graph Embeddings

Index terms have been assigned to the content through auto-classification.

Recommendations

Federated knowledge graph completion via embedding-contrastive learning
Abstract
Recently, the research about knowledge graphs (KGs) which contain a large number of triples, has gained massive attention. Many knowledge graph embedding methods are proposed to tackle the knowledge graph completion task. In real ...
MulDE: Multi-teacher Knowledge Distillation for Low-dimensional Knowledge Graph Embeddings
WWW '21: Proceedings of the Web Conference 2021

Link prediction based on knowledge graph embeddings (KGE) aims to predict new triples to automatically construct knowledge graphs (KGs). However, recent KGE models achieve performance improvements by excessively increasing the embedding dimensions, ...
Contrastive learning from label distribution: A case study on text classification
Highlights
- The proposed method learns under the supervision of the predicted label distribution.
Abstract
State-of-the-art text classification models are dominated by deep neural networks, but they still struggle to the issue of poor generalization ability when using cross entropy loss for training. One of the reasons is the training ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Natural Science Foundation in China
Australian Research Council (ARC) Discovery Project
Fundamental Research Fund for Central University
Australian Research Council (ARC) Future Fellowship

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
445
Total Downloads

Downloads (Last 12 months)99
Downloads (Last 6 weeks)3

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HZhang JMolybog IChua TNgo CKa-Wei Lee RKumar RLauw H(2024)HaSa: Hardness and Structure-Aware Contrastive Knowledge Graph EmbeddingProceedings of the ACM on Web Conference 202410.1145/3589334.3645564(2116-2127)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645564
Jiang LYan GLuo HChang W(2023)Improved Collaborative Recommendation Model: Integrating Knowledge Embedding and Graph Contrastive LearningElectronics10.3390/electronics1220423812:20(4238)Online publication date: 13-Oct-2023
https://doi.org/10.3390/electronics12204238
Zhong LWu JLi QPeng HWu X(2023)A Comprehensive Survey on Automatic Knowledge Graph ConstructionACM Computing Surveys10.1145/361829556:4(1-62)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3618295
Tan ZChen ZFeng SZhang QZheng QLi JLuo M(2023)KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph CompletionProceedings of the ACM Web Conference 202310.1145/3543507.3583412(2548-2559)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583412
Bastos ASingh KNadgeri AHoffart JSingh MSuzumura T(2023)Can Persistent Homology provide an efficient alternative for Evaluation of Knowledge Graph Completion Methods?Proceedings of the ACM Web Conference 202310.1145/3543507.3583308(2455-2466)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583308
Shang BZhao YWang DLiu JChen HDuh WHuang HKato MMothe JPoblete B(2023)Relation-Aware Multi-Positive Contrastive Knowledge Graph Completion with Embedding Dimension ScalingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591756(878-888)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591756
Cui HPeng TXiao FHan JHan RLiu L(2023)Incorporating anticipation embedding into reinforcement learning framework for multi-hop knowledge graph question answeringInformation Sciences: an International Journal10.1016/j.ins.2022.11.042619:C(745-761)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.ins.2022.11.042
Wu YChen YYin ZDing WKing I(2023)A survey on graph embedding techniques for biomedical dataInformation Fusion10.1016/j.inffus.2023.101909100:COnline publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1016/j.inffus.2023.101909
Lin DChen ZZhao LWang K(2022)Multi-label Aerial Image Classification Based on Image-Specific Concept Graphs2022 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP46576.2022.9897476(121-125)Online publication date: 16-Oct-2022
https://doi.org/10.1109/ICIP46576.2022.9897476

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents