Abstract
The ubiquity of the contemporary language understanding tasks gives relevance to the development of generalized, yet highly efficient models that utilize all knowledge, provided by the data source. In this work, we present SocialBERT - the first model that uses knowledge about the author’s position in the network during text analysis. We investigate possible models for learning social network information and successfully inject it into the baseline BERT model. The evaluation shows that embedding this information maintains a good generalization, with an increase in the quality of the probabilistic model for the given author up to 7.5%. The proposed model has been trained on the majority of groups for the chosen social network, and still able to work with previously unknown groups. The obtained model is available for download and use in applied tasks (https://github.com/karpovilia/SocialBert).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Rosenthal, S., Farra, N., Nakov, P.: SemEval-2017 task 4: sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 502–518. Association for Computational Linguistics, Vancouver, Canada (2017)
Pontiki, M., et al.: SemEval-2016 task 5: aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 19–30. Association for Computational Linguistics, San Diego, California (2016)
Sorokin, A., et al.: MorphoRuEval-2017: an evaluation track for the automatic morphological analysis methods for Russian. Komp’yuternaya lingvistika i intellektual’nyye tekhnologii 1, 297–313 (2017)
Crystal, D.: Language and the Internet. Cambridge University Press, Cambridge (2006)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Pre-training of deep bidirectional transformers for language understanding, Bert (2019)
Nguyen, D.Q., Vu, T., Nguyen, A.T.: BERTweet: a pre-trained language model for English tweets. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 9–14. Association for Computational Linguistics (2020)
Marsden, P.V.: Homogeneity in confiding relations. Soc. Netw. 10(1), 57–76 (1988)
Adamic, L.A., Glance, N.: The political blogosphere and the 2004 U.S. election: Divided they blog. In: Proceedings of the 3rd International Workshop on Link Discovery, LinkKDD’05, pp. 36–43. Association for Computing Machinery, New York (2005)
Rieder, B.: Studying facebook via data extraction: the Netvizz application. In: Proceedings of the 5th Annual ACM Web Science Conference, WebSci’13, pp. 346–355. Association for Computing Machinery, New York(2013)
Keikha, M.M., Rahgozar, M., Asadpour, M.: Community aware random walk for network embedding. Knowl. Based Syst. 148, 47–54 (2018)
Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2014)
Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks (2016)
Zhang, Z., Cui, P., Wang, X., Pei, J., Yao, X., Zhu, W.: Arbitrary-order proximity preserved network embedding, pp. 2778–2786. Association for Computing Machinery, New York (2018)
Zhang, J., Zhang, H., Xia, C., Sun, L.: Graph-bert: only attention is needed for learning graph representations (2020)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks (2018)
Gururangan, S., et al.: Don’t stop pretraining: adapt language models to domains and tasks (2020)
Han, X., Eisenstein, J.: Unsupervised domain adaptation of contextualized embeddings for sequence labeling (2019)
Yogatama, D., et al.: Learning and evaluating general linguistic intelligence (2019)
Mghabbar, I., Ratnamogan, P.: Building a multi-domain neural machine translation model using knowledge distillation (2020)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015)
Lu, Z., Du, P., Nie, J.-Y.: VGCN-BERT: augmenting BERT with graph embedding for text classification. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12035, pp. 369–382. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45439-5_25
Lauscher, A., Majewska, O., Ribeiro, L.F., Gurevych, I., Rozanov, N., Glavaš, G.: Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers. In: Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 43–49. Association for Computational Linguistics (2020)
Kuratov, Y., Arkhipov, M.: Adaptation of deep bidirectional multilingual transformers for Russian language (2019)
Acknowledgements
The article was prepared within the framework of the HSE University Basic Research Program and through computational resources of HPC facilities provided by NRU HSE.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Karpov, I., Kartashev, N. (2022). SocialBERT – Transformers for Online Social Network Language Modelling. In: Burnaev, E., et al. Analysis of Images, Social Networks and Texts. AIST 2021. Lecture Notes in Computer Science, vol 13217. Springer, Cham. https://doi.org/10.1007/978-3-031-16500-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-16500-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16499-6
Online ISBN: 978-3-031-16500-9
eBook Packages: Computer ScienceComputer Science (R0)