Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3548606.3560554acmconferencesArticle/Chapter ViewAbstractPublication PagesccsConference Proceedingsconference-collections
research-article
Open access

Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets

Published: 07 November 2022 Publication History

Abstract

We introduce a new class of attacks on machine learning models. We show that an adversary who can poison a training dataset can cause models trained on this dataset to leak significant private details of training points belonging to other parties. Our active inference attacks connect two independent lines of work targeting the integrity and privacy of machine learning training data.
Our attacks are effective across membership inference, attribute inference, and data extraction. For example, our targeted attacks can poison <0.1% of the training dataset to boost the performance of inference attacks by 1 to 2 orders of magnitude. Further, an adversary who controls a significant fraction of the training data (e.g., 50%) can launch untargeted attacks that enable 8× more precise inference on all other users' otherwise-private data points.
Our results cast doubts on the relevance of cryptographic privacy guarantees in multiparty computation protocols for machine learning, if parties can arbitrarily select their share of training data.

References

[1]
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security, 2016.
[2]
Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 13(5):1333--1345, 2017.
[3]
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In USENIX Security Symposium, pages 1505--1521, 2021.
[4]
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938--2948. PMLR, 2020.
[5]
Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency, 2021.
[6]
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pages 634--643. PMLR, 2019.
[7]
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In International Conference on Machine Learning, 2012.
[8]
Daniel Bleichenbacher. Chosen ciphertext attacks against protocols based on the RSA encryption standard PKCS#1. In Annual International Cryptology Conference, pages 1--12. Springer, 1998.
[9]
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning is not private. arXiv preprint arXiv:2112.02918, 2021.
[10]
Dan Boneh and Victor Shoup. A Graduate Course in Applied Cryptography. http://toc.cryptobook.us/, 2020.
[11]
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In IEEE Symposium on Security and Privacy, pages 1897--1914. IEEE, 2022.
[12]
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022.
[13]
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The se- cret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, pages 267--284, 2019.
[14]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, 2021.
[15]
Moses Charikar, Jacob Steinhardt, and Gregory Valiant. Learning from untrusted data. In ACM SIGACT Symposium on Theory of Computing, pages 47--60, 2017.
[16]
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pages 1596--1606. PMLR, 2019.
[17]
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, pages 265--284. Springer, 2006.
[18]
Vitaly Feldman. Does learning require memorization? A short tale about a long tail. In ACM SIGACT Symposium on Theory of Computing, pages 954--959, 2020.
[19]
Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted transformers breach privacy in federated learning for language models. arXiv preprint arXiv:2201.12675, 2022.
[20]
Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. Advances in Neural Information Processing Systems, 34, 2021.
[21]
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM SIGSAC Conference on Computer and Communications Security, pages 1322--1333, 2015.
[22]
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In USENIX Security Symposium, 2014.
[23]
Jonas Geiping, Liam Fowl, Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In International Conference on Learning Representations, 2021.
[24]
Yoel Gluck, Neal Harris, and Angelo Prado. Breach: reviving the crime attack. http://breachattack.com, 2013.
[25]
Oded Goldreich, Silvio Micali, and Avi Wigderson. How to play any mental game, or a completeness theorem for protocols with honest majority. In ACM SIGACT Symposium on Theory of Computing, 1987.
[26]
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7, 2019.
[27]
Neal Gupta, W Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John Dickerson. Strong baseline defenses against clean-label poisoning attacks. https://openreview.net/forum?id=B1xgv0NtwH, 2019.
[28]
Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz. Deep models under the GAN: Information leakage from collaborative deep learning. In ACM SIGSAC Conference on Computer and Communications Security, pages 603--618, 2017.
[29]
Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John Pearson, Dietrich Stephan, Stanley Nelson, and David Craig. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS genetics, 2008.
[30]
Lin-Shung Huang, Zack Weinberg, Chris Evans, and Collin Jackson. Protecting browsers from cross-origin CSS attacks. In ACM SIGSAC Conference on Computer and Communications Security, pages 619--629, 2010.
[31]
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In IEEE Symposium on Security and Privacy, 2018.
[32]
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? Advances in Neural Information Processing Systems, 33:22205-22216, 2020.
[33]
Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu. Revisiting membership inference under realistic assumptions. In Proceedings on Privacy Enhancing Technologies, 2021.
[34]
Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. MemGuard: Defending against black-box membership inference attacks via adversarial examples. In ACM SIGSAC Conference on Computer and Communications Security, pages 259--274, 2019.
[35]
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210, 2021.
[36]
John Kelsey. Compression and information leakage of plaintext. In International Workshop on Fast Software Encryption, pages 263--276. Springer, 2002.
[37]
Ronny Kohavi and Barry Becker. UCI machine learning repository: Adult data set. https://archive.ics.uci.edu/ml/machine-learning-databases/adult, 1996.
[38]
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
[39]
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Network and Distributed System Security Symposium, 2018.
[40]
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In 2017 IEEE International Conference on Computer Design (ICCD), pages 45--48. IEEE, 2017.
[41]
Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In IEEE European Symposium on Security and Privacy, pages 521--534. IEEE, 2020.
[42]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[43]
S. Mahloujifar, E. Ghosh, and M. Chase. Property inference from poisoning. In IEEE Symposium on Security and Privacy, pages 1569--1569, Los Alamitos, CA, USA, may 2022. IEEE Computer Society.
[44]
Mani Malek Esmaeili, Ilya Mironov, Karthik Prasad, Igor Shilov, and Florian Tramer. Antipodes of label differential privacy: PATE and ALIBI. Advances in Neural Information Processing Systems, 34, 2021.
[45]
Shagufta Mehnaz, Sayanton V Dibbo, Ehsanul Kabir, Ninghui Li, and Elisa Bertino. Are your sensitive attributes private? Novel model inversion attribute inference attacks on classification models. In USENIX Security Symposium, 2022.
[46]
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Ex- ploiting unintended feature leakage in collaborative learning. In IEEE Symposium on Security and Privacy, pages 691--706. IEEE, 2019.
[47]
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[48]
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929, 2022.
[49]
Payman Mohassel and Peter Rindal. ABY3: A mixed protocol framework for machine learning. In ACM SIGSAC Conference on Computer and Communications Security, pages 35--52, 2018.
[50]
Payman Mohassel and Yupeng Zhang. SecureML: A system for scalable privacy- preserving machine learning. In IEEE Symposium on Security and Privacy, pages 19--38. IEEE, 2017.
[51]
Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In ACM Workshop on Artificial Intelligence and Security, pages 27--38, 2017.
[52]
Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In ACM SIGSAC Conference on Computer and Communications Security, pages 634--646, 2018.
[53]
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In IEEE Symposium on Security and Privacy, pages 739--753. IEEE, 2019.
[54]
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In IEEE Symposium on Security and Privacy, pages 866--882. IEEE, 2021.
[55]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748--8763. PMLR, 2021.
[56]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019.
[57]
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data. arXiv preprint arXiv:2009.10031, 2020.
[58]
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, 2019.
[59]
Hadi Salman, Andrew Ilyas, Logan Engstrom, Sai Vemprala, Aleksander Madry, and Ashish Kapoor. Unadversarial examples: Designing objects for robust vision. Advances in Neural Information Processing Systems, 34, 2021.
[60]
Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You auto- complete me: Poisoning vulnerabilities in neural code completion. In USENIX Security Symposium, pages 1559--1575, 2021.
[61]
Ali Shafahi, Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! Targeted clean-label poisoning attacks on neural networks. Advances in Neural Information Processing Systems, 2018.
[62]
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, pages 3--18. IEEE, 2017.
[63]
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In ACM SIGSAC Conference on Computer and Communications Security, pages 587--601, 2017.
[64]
Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras. When does machine learning FAIL? Generalized transferability for evasion and poisoning attacks. In USENIX Security Symposium, 2018.
[65]
Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. Understanding unintended memorization in language models under federated learning. In Workshop on Privacy in Natural Language Processing, 2021.
[66]
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in Neural Information Processing Systems, 31, 2018.
[67]
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771, 2019.
[68]
Serge Vaudenay. Security flaws induced by CBC padding-applications to SSL, IPSEC, WTLS... In International Conference on the Theory and Applications of Cryptographic Techniques, pages 534--545. Springer, 2002.
[69]
Sameer Wagh, Divya Gupta, and Nishanth Chandran. SecureNN: 3-party secure computation for neural network training. Proceedings on Privacy Enhancing Technologies, 2019(3):26--49, 2019.
[70]
Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations, 2022.
[71]
Yuxin Wen, Jonas A. Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein. Fishing for user data in large-batch federated learning via gradient magnification. In International Conference on Machine Learning, pages 23668--23684. PMLR, 2022.
[72]
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
[73]
Andrew C Yao. Protocols for secure computations. In 23rd annual Symposium on Foundations of Computer Science, pages 160--164. IEEE, 1982.
[74]
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In ACM SIGSAC Conference on Computer and Communications Security, 2022.
[75]
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE Computer Security Foundations Symposium, pages 268--282. IEEE, 2018.
[76]
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
[77]
Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. In ACM SIGSAC Conference on Computer and Communications Security, pages 363--375, 2020.
[78]
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In International Conference on Machine Learning, pages 7614--7623. PMLR, 2019.

Cited By

View all
  • (2024)Privacy-Preserving Deep Learning Framework Based on Restricted Boltzmann Machines and Instance Reduction AlgorithmsApplied Sciences10.3390/app1403122414:3(1224)Online publication date: 1-Feb-2024
  • (2024)Where Have You Been? A Study of Privacy Risk for Point-of-Interest RecommendationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671758(175-186)Online publication date: 25-Aug-2024
  • (2024)Digital transformation: A multidisciplinary perspective and future research agendaInternational Journal of Consumer Studies10.1111/ijcs.1301548:2Online publication date: 13-Feb-2024
  • Show More Cited By

Index Terms

  1. Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CCS '22: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security
      November 2022
      3598 pages
      ISBN:9781450394505
      DOI:10.1145/3548606
      This work is licensed under a Creative Commons Attribution International 4.0 License.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 07 November 2022

      Check for updates

      Author Tags

      1. machine learning
      2. membership inference
      3. poisoning
      4. privacy

      Qualifiers

      • Research-article

      Conference

      CCS '22
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

      Upcoming Conference

      CCS '24
      ACM SIGSAC Conference on Computer and Communications Security
      October 14 - 18, 2024
      Salt Lake City , UT , USA

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)1,397
      • Downloads (Last 6 weeks)84
      Reflects downloads up to 30 Aug 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Privacy-Preserving Deep Learning Framework Based on Restricted Boltzmann Machines and Instance Reduction AlgorithmsApplied Sciences10.3390/app1403122414:3(1224)Online publication date: 1-Feb-2024
      • (2024)Where Have You Been? A Study of Privacy Risk for Point-of-Interest RecommendationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671758(175-186)Online publication date: 25-Aug-2024
      • (2024)Digital transformation: A multidisciplinary perspective and future research agendaInternational Journal of Consumer Studies10.1111/ijcs.1301548:2Online publication date: 13-Feb-2024
      • (2024)Deepmarking: Leveraging Adversarial Noise for Membership Inference Attacks2024 IEEE International Conference on Computational Photography (ICCP)10.1109/ICCP61108.2024.10644615(1-10)Online publication date: 22-Jul-2024
      • (2024)Unintended Memorization in Large ASR Models, and How to Mitigate ItICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10446083(4655-4659)Online publication date: 14-Apr-2024
      • (2024)REN-A.I.: A Video Game for AI Security Education Leveraging Episodic MemoryIEEE Access10.1109/ACCESS.2024.337769912(47359-47372)Online publication date: 2024
      • (2024)The Importance of Corporate Digital Responsibility in a Digital Service WorldThe Impact of Digitalization on Current Marketing Strategies10.1108/978-1-83753-686-320241011(183-193)Online publication date: 14-Mar-2024
      • (2024)Privacy-preserving edge Federated Learning for intelligent mobile-health systemsFuture Generation Computer Systems10.1016/j.future.2024.07.035Online publication date: Jul-2024
      • (2024)Private Information Leakage in LLMsLarge Language Models in Cybersecurity10.1007/978-3-031-54827-7_7(75-79)Online publication date: 12-Apr-2024
      • (2024)Towards Privacy Preserving LLMs TrainingLarge Language Models in Cybersecurity10.1007/978-3-031-54827-7_19(175-179)Online publication date: 12-Apr-2024
      • Show More Cited By

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Get Access

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media