Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Memorization of Named Entities in Fine-Tuned BERT Models

  • Conference paper
  • First Online:
Machine Learning and Knowledge Extraction (CD-MAKE 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14065))

  • 783 Accesses

Abstract

Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that have been trained on datasets, which contain personal and privacy sensitive information. In our study, we investigate the extent of named entity memorization in fine-tuned BERT models. We use single-label text classification as representative downstream task and employ three different fine-tuning setups in our experiments, including one with Differentially Privacy (DP). We create a large number of text samples from the fine-tuned BERT models utilizing a custom sequential sampling strategy with two prompting strategies. We search in these samples for named entities and check if they are also present in the fine-tuning datasets. We experiment with two benchmark datasets in the domains of emails and blogs. We show that the application of DP has a detrimental effect on the text generation capabilities of BERT. Furthermore, we show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only. This suggests that BERT is unlikely to emit personal or privacy sensitive named entities. Overall, our results are important to understand to what extent BERT-based services are prone to training data extraction attacks (Source code and datasets are available at: https://github.com/drndr/bert_ent_attack. An extended version of this paper can be also found on arXiv [12]).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://commoncrawl.org.

References

  1. Abadi, M., et al.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308–318 (2016)

    Google Scholar 

  2. Acar, A., Aksu, H., Uluagac, A.S., Conti, M.: A survey on homomorphic encryption schemes: theory and implementation. ACM Comput. Surv. (CSUR) 51(4), 1–35 (2018)

    Article  Google Scholar 

  3. Alsentzer, E., et al.: Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 (2019)

  4. Aono, Y., Hayashi, T., Wang, L., Moriai, S., et al.: Privacy-preserving deep learning via additively homomorphic encryption. IEEE Trans. Inf. Forensics Secur. 13(5), 1333–1345 (2017)

    Google Scholar 

  5. Beltagy, I., Lo, K., Cohan, A.: SciBERT: a pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)

  6. Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)

    Google Scholar 

  7. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., Song, D.: The secret sharer: evaluating and testing unintended memorization in neural networks. In: 28th USENIX Security Symposium (USENIX Security 2019), pp. 267–284 (2019)

    Google Scholar 

  8. Carlini, N., et al.: Extracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 2021), pp. 2633–2650 (2021)

    Google Scholar 

  9. Davody, A., Adelani, D.I., Kleinbauer, T., Klakow, D.: Robust differentially private training of deep neural networks. arXiv preprint arXiv:2006.10919 (2020)

  10. De Cristofaro, E.: An overview of privacy in machine learning. arXiv preprint arXiv:2005.08679 (2020)

  11. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  12. Diera, A., Lell, N., Garifullina, A., Scherp, A.: A study on extracting named entities from fine-tuned vs. differentially private fine-tuned BERT models. CoRR abs/2212.03749 (2022). https://doi.org/10.48550/arXiv.2212.03749

  13. Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., Smith, N.: Fine-tuning pretrained language models: weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 (2020)

  14. Dupuy, C., Arava, R., Gupta, R., Rumshisky, A.: An efficient DP-SGD mechanism for large scale NLP models. arXiv preprint arXiv:2107.14586 (2021)

  15. Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., Naor, M.: Our data, ourselves: privacy via distributed noise generation. In: Vaudenay, S. (ed.) EUROCRYPT 2006. LNCS, vol. 4004, pp. 486–503. Springer, Heidelberg (2006). https://doi.org/10.1007/11761679_29

    Chapter  Google Scholar 

  16. Dwork, C., Roth, A., et al.: The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. 9(3–4), 211–407 (2014)

    MathSciNet  MATH  Google Scholar 

  17. Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322–1333 (2015)

    Google Scholar 

  18. Freitag, M., Al-Onaizan, Y.: Beam search strategies for neural machine translation. In: Proceedings of the First Workshop on Neural Machine Translation, pp. 56–60 (2017)

    Google Scholar 

  19. Galke, L., Scherp, A.: Bag-of-words vs. graph vs. sequence in text classification: questioning the necessity of text-graphs and the surprising strength of a wide MLP. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, 22–27 May 2022, pp. 4038–4051. Association for Computational Linguistics (2022)

    Google Scholar 

  20. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. 6, 721–741 (1984)

    Article  MATH  Google Scholar 

  21. Ghazvininejad, M., Levy, O., Liu, Y., Zettlemoyer, L.: Mask-predict: parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324 (2019)

  22. Gilad-Bachrach, R., Dowlin, N., Laine, K., Lauter, K., Naehrig, M., Wernsing, J.: CryptoNets: applying neural networks to encrypted data with high throughput and accuracy. In: International Conference on Machine Learning, pp. 201–210. PMLR (2016)

    Google Scholar 

  23. Hassan, F., Domingo-Ferrer, J., Soria-Comas, J.: Anonymization of unstructured data via named-entity recognition. In: Torra, V., Narukawa, Y., Aguiló, I., González-Hidalgo, M. (eds.) MDAI 2018. LNCS (LNAI), vol. 11144, pp. 296–305. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00202-2_24

    Chapter  Google Scholar 

  24. He, P., Liu, X., Gao, J., Chen, W.: DeBERTa: decoding-enhanced BERT with disentangled attention. arXiv preprint arXiv:2006.03654 (2020)

  25. Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019)

  26. Honnibal, M., Montani, I., Van Landeghe, S., Boyd, A.: spaCy: industrial-strength natural language processing in python (2022). https://zenodo.org/record/121230

  27. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 (2018)

  28. Klimt, B., Yang, Y.: Introducing the Enron corpus. In: CEAS 2004 - First Conference on Email and Anti-Spam, 30–31 July 2004, Mountain View, California, USA (2004)

    Google Scholar 

  29. Lee, J., Tang, R., Lin, J.: What would Elsa do? Freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090 (2019)

  30. Lehman, E., Jain, S., Pichotta, K., Goldberg, Y., Wallace, B.C.: Does BERT pretrained on clinical notes reveal sensitive data? arXiv preprint arXiv:2104.07762 (2021)

  31. Lhoest, Q., et al.: Datasets: a community library for natural language processing. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175–184. Association for Computational Linguistics, Online and Punta Cana (2021)

    Google Scholar 

  32. Liu, B., Ding, M., Shaham, S., Rahayu, W., Farokhi, F., Lin, Z.: When machine learning meets privacy: a survey and outlook. ACM Comput. Surv. (CSUR) 54(2), 1–36 (2021)

    Article  Google Scholar 

  33. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  34. Liu, Z., Winata, G.I., Madotto, A., Fung, P.: Exploring fine-tuning techniques for pre-trained cross-lingual models via continual learning. arXiv preprint arXiv:2004.14218 (2020)

  35. Mao, H.H.: A survey on self-supervised pre-training for sequential transfer learning in neural networks. arXiv preprint arXiv:2007.00800 (2020)

  36. McMahan, H.B., Moore, E., Ramage, D., y Arcas, B.A.: Federated learning of deep networks using model averaging. arXiv preprint arXiv:1602.05629 (2016)

  37. Mireshghallah, F., Taram, M., Vepakomma, P., Singh, A., Raskar, R., Esmaeilzadeh, H.: Privacy in deep learning: a survey. arXiv preprint arXiv:2004.12254 (2020)

  38. Nadeau, D., Sekine, S.: A survey of named entity recognition and classification. Lingvisticae Invest. 30(1), 3–26 (2007)

    Article  Google Scholar 

  39. Oh, S.J., Schiele, B., Fritz, M.: Towards reverse-engineering black-box neural networks. In: Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., Müller, K.-R. (eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS (LNAI), vol. 11700, pp. 121–144. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28954-6_7

    Chapter  Google Scholar 

  40. Ouyang, L., et al.: Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022)

  41. Parisot, M.P., Pejo, B., Spagnuelo, D.: Property inference attacks on convolutional neural networks: influence and implications of target model’s complexity. arXiv preprint arXiv:2104.13061 (2021)

  42. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019)

  43. Rigaki, M., Garcia, S.: A survey of privacy attacks in machine learning. arXiv preprint arXiv:2007.07646 (2020)

  44. Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in BERTology: what we know about how BERT works. Trans. Assoc. Comput. Linguist. 8, 842–866 (2020)

    Article  Google Scholar 

  45. Rubinstein, B.I., Bartlett, P.L., Huang, L., Taft, N.: Learning in a large function space: privacy-preserving mechanisms for SVM learning. arXiv preprint arXiv:0911.5708 (2009)

  46. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

  47. Schler, J., Koppel, M., Argamon, S., Pennebaker, J.: Effects of age and gender on blogging in proceedings of 2006 AAAI spring symposium on computational approaches for analyzing weblogs. In: Proceedings of 2006 AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs (2006)

    Google Scholar 

  48. Sharir, O., Peleg, B., Shoham, Y.: The cost of training NLP models: a concise overview. arXiv preprint arXiv:2004.08900 (2020)

  49. Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 3–18. IEEE (2017)

    Google Scholar 

  50. Singhofer, F., Garifullina, A., Kern, M., Scherp, A.: A novel approach on the joint de-identification of textual and relational data with a modified Mondrian algorithm. In: DocEng 2021: ACM Symposium on Document Engineering 2021, 24–27 August 2021, pp. 14:1–14:10. ACM (2021)

    Google Scholar 

  51. Sun, W., Khan, H., Guenon des Mesnards, N., Rubino, M., Arkoudas, K.: Unfreeze with care: space-efficient fine-tuning of semantic parsing models. In: Proceedings of the ACM Web Conference 2022, pp. 999–1007 (2022)

    Google Scholar 

  52. Sweeney, L.: k-anonymity: a model for protecting privacy. Internat. J. Uncertain. Fuzziness Knowl.-Based Syst. 10(05), 557–570 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  53. Thakkar, O., Ramaswamy, S., Mathews, R., Beaufays, F.: Understanding unintended memorization in federated learning. arXiv preprint arXiv:2006.07490 (2020)

  54. Tramèr, F., Zhang, F., Juels, A., Reiter, M.K., Ristenpart, T.: Stealing machine learning models via prediction APIs. In: 25th USENIX security symposium (USENIX Security 2016), pp. 601–618 (2016)

    Google Scholar 

  55. Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)

    Google Scholar 

  56. Wallace, E., Wang, Y., Li, S., Singh, S., Gardner, M.: Do NLP models know numbers? Probing numeracy in embeddings. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5307–5315 (2019)

    Google Scholar 

  57. Wang, A., Cho, K.: BERT has a mouth, and it must speak: BERT as a Markov random field language model. arXiv preprint arXiv:1902.04094 (2019)

  58. Wang, B., Gong, N.Z.: Stealing hyperparameters in machine learning. In: 2018 IEEE Symposium on Security and Privacy (SP), pp. 36–52. IEEE (2018)

    Google Scholar 

  59. Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)

    Google Scholar 

  60. Xu, Y., Ma, T., Tang, M., Tian, W.: A survey of privacy preserving data publishing using generalization and suppression. Appl. Math. Inf. Sci. 8(3), 1103 (2014)

    Article  Google Scholar 

  61. Yousefpour, A., et al.: Opacus: user-friendly differential privacy library in pytorch. arXiv preprint arXiv:2109.12298 (2021)

  62. Yu, D., et al.: Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500 (2021)

  63. Zanella-Beguelin, S., et al.: Analyzing information leakage of updates to natural language models. In: Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. 363–375 (2020)

    Google Scholar 

  64. Zhang, H., Song, H., Li, S., Zhou, M., Song, D.: A survey of controllable text generation using transformer-based pre-trained language models. arXiv preprint arXiv:2201.05337 (2022)

  65. Zhang, T., He, Z., Lee, R.B.: Privacy-preserving machine learning through data obfuscation. arXiv preprint arXiv:1807.01860 (2018)

  66. Zhu, T., Ye, D., Wang, W., Zhou, W., Yu, P.: More than privacy: applying differential privacy in key areas of artificial intelligence. IEEE Trans. Knowl. Data Eng. 34, 2824–2843 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Ulm University, Ulm, Germany

    Andor Diera, Nicolas Lell & Ansgar Scherp

  2. BT, Ipswich, UK

    Aygul Garifullina

Authors
  1. Andor Diera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicolas Lell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aygul Garifullina
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ansgar Scherp
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andor Diera .

Editor information

Editors and Affiliations

  1. University of Natural Resources and Life, Vienna, Austria

    Andreas Holzinger

  2. St. Pölten University of Applied Science, St. Pölten, Austria

    Peter Kieseberg

  3. University of Milano-Bicocca, Milan, Italy

    Federico Cabitza

  4. University of Milano-Bicocca, Milan, Italy

    Andrea Campagner

  5. TU Wien, Vienna, Austria

    A Min Tjoa

  6. SBA Research, Vienna, Austria

    Edgar Weippl

Rights and permissions

Reprints and permissions

Copyright information

© 2023 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Diera, A., Lell, N., Garifullina, A., Scherp, A. (2023). Memorization of Named Entities in Fine-Tuned BERT Models. In: Holzinger, A., Kieseberg, P., Cabitza, F., Campagner, A., Tjoa, A.M., Weippl, E. (eds) Machine Learning and Knowledge Extraction. CD-MAKE 2023. Lecture Notes in Computer Science, vol 14065. Springer, Cham. https://doi.org/10.1007/978-3-031-40837-3_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-40837-3_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-40836-6

  • Online ISBN: 978-3-031-40837-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation