Cited By
View all- Wu NLin XLu JZhang FChen WTang JXiao J(2024)Byzantine-Robust Multimodal Federated Learning Framework for Intelligent Connected VehicleElectronics10.3390/electronics1318363513:18(3635)Online publication date: 12-Sep-2024
BadSampler: Harnessing the Power of Catastrophic Forgetting to Poison Byzantine-robust Federated Learning
Authors: 
Yi Liu, Cong Wang, Xingliang YuanAuthors Info & Claims
Yi Liu, Cong Wang, Xingliang YuanAuthors Info & ClaimsPages 1944 - 1955
Abstract
Federated Learning (FL) is susceptible to poisoning attacks, wherein compromised clients manipulate the global model by modifying local datasets or sending manipulated model updates. Experienced defenders can readily detect and mitigate the poisoning effects of malicious behaviors using Byzantine-robust aggregation rules. However, the exploration of poisoning attacks in scenarios where such behaviors are absent remains largely unexplored for Byzantine-robust FL. This paper addresses the challenging problem of poisoning Byzantine-robust FL by introducing catastrophic forgetting. To fill this gap, we first formally define generalization error and establish its connection to catastrophic forgetting, paving the way for the development of a clean-label data poisoning attack named BadSampler. This attack leverages only clean-label data (i.e., without poisoned data) to poison Byzantine-robust FL and requires the adversary to selectively sample training data with high loss to feed model training and maximize the model's generalization error. We formulate the attack as an optimization problem and present two elegant adversarial sampling strategies, Top-k sampling, and meta-sampling, to approximately solve it. Additionally, our formal error upper bound and time complexity analysis demonstrate that our design can preserve attack utility with high efficiency. Extensive evaluations on two real-world datasets illustrate the effectiveness and performance of our proposed attacks.
Supplemental Material
MP4 File - rtfp1265-2min-promo
We propose BadSampler, which can poison state-of-the-art Byzantine-robust federated learning without requiring additional knowledge and unrealistic attack assumptions.
- Download
- 30.61 MB
References
[1]
Kareem Amin, Alex Kulesza, Andres Munoz, and Sergei Vassilvtiskii. 2019. Bounding user contributions: A bias-variance trade-off in differential privacy. In Proc. of ICML.
[2]
Sana Awan, Bo Luo, and Fengjun Li. 2021. CONTRA: Defending against Poisoning Attacks in Federated Learning. In Proc. of ESORICS.
[3]
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. 2020. How to backdoor federated learning. In Proc. of AISTATS.
[4]
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. 2019. Analyzing federated learning through an adversarial lens. In Proc. of ICML.
[5]
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. 2017. Machine learning with adversaries: Byzantine tolerant gradient descent. In Proc. of NeurIPS.
[6]
Peva Blanchard, Rachid Guerraoui, Julien Stainer, et al. 2017. Machine learning with adversaries: Byzantine tolerant gradient descent. In Proc. of NeurIPS.
[7]
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konevcn?, Stefano Mazzocchi, Brendan McMahan, et al. 2019. Towards federated learning at scale: System design. In Proc. of MLSys.
[8]
Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. 2021. FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping. In Proc. of NDSS.
[9]
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. 2020. Local model poisoning attacks to Byzantine-robust federated learning. In Proc. of USENIX Security.
[10]
Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. 2020. The limitations of federated learning in sybil settings. In Proc. of RAID.
[11]
Stuart Geman, Elie Bienenstock, and René Doursat. 1992. Neural networks and the bias/variance dilemma. Neural computation, Vol. 4, 1 (1992), 1--58.
[12]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. of ICML.
[13]
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. 2018. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In Proc. of IEEE S&P.
[14]
Malhar S Jere, Tyler Farnan, and Farinaz Koushanfar. 2020. A taxonomy of attacks on federated learning. IEEE Security & Privacy, Vol. 19, 2 (2020), 20--28.
[15]
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, Vol. 114, 13 (2017), 3521--3526.
[16]
A Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Master's thesis, University of Tront (2009).
[17]
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In Proc. of NeurIPS.
[18]
Liping Li, Wei Xu, Tianyi Chen, Georgios B Giannakis, and Qing Ling. 2019. RSA: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets. In Proc. of AAAI.
[19]
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. 2022. Federated learning on non-iid data silos: An experimental study. In Proc. of ICDE.
[20]
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, Vol. 37, 3 (2020), 50--60.
[21]
Feng Lin, Qing Ling, and Zhiwei Xiong. 2019. Byzantine-resilient distributed large-scale matrix completion. In Proc. of ICASSP.
[22]
Chen Liu, Zhichao Huang, Mathieu Salzmann, Tong Zhang, and Sabine Süsstrunk. 2021. On the impact of hard adversarial instances on overfitting in adversarial training. arXiv preprint arXiv:2112.07324 (2021).
[23]
Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tie-Yan Liu. 2020. Self-paced ensemble for highly imbalanced massive data classification. In Proc. of ICDE.
[24]
Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. 2020. MESA: boost ensemble imbalanced learning with meta-sampler. In Proc. of NeurIPS.
[25]
Yunlong Mao, Xinyu Yuan, Xinyang Zhao, and Sheng Zhong. 2021. Romoa: Robust model aggregation for the resistance of federated learning to model poisoning attacks. In Proc. of ESORICS.
[26]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proc. of AISTATS.
[27]
Matias Mendieta and et al. 2022. Local learning matters: Rethinking data heterogeneity in federated learning. In Proc. of CVPR.
[28]
Chuizheng Meng, Sirisha Rambhatla, and Yan Liu. 2021. Cross-node federated graph neural network for spatio-temporal data modeling. In Proc. of KDD.
[29]
Jung Wuk Park, Dong-Jun Han, Minseok Choi, and Jaekyun Moon. 2021. Sageflow: Robust Federated Learning against Both Stragglers and Adversaries. In Proc. of NeurIPS.
[30]
Md Mahmudur Rahman and Sanjay Purushotham. 2023. FedPseudo: Privacy-Preserving Pseudo Value-Based Deep Learning Models for Federated Survival Analysis. In Proc. of KDD.
[31]
Phillip Rieger, Thien Duc Nguyen, Markus Miettinen, and Ahmad-Reza Sadeghi. 2022. DeepSight: Mitigating Backdoor Attacks in Federated Learning Through Deep Model Inspection. In Proc. of NDSS.
[32]
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proc. of NeurIPS.
[33]
Virat Shejwalkar and Amir Houmansadr. 2021. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning. In Proc. of NDSS.
[34]
Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. 2022. Back to the drawing board: A critical evaluation of poisoning attacks on federated learning. In Proc. of IEEE S&P.
[35]
Shiqi Shen, Shruti Tople, and Prateek Saxena. 2016. Auror: Defending against poisoning attacks in collaborative deep learning systems. In Proc. of ACSAC.
[36]
Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross Anderson. 2021. Manipulating SGD with data ordering attacks. In Proc. of NeurIPS.
[37]
Samuel L Smith, Benoit Dherin, David Barrett, and Soham De. 2021. On the Origin of Implicit Regularization in Stochastic Gradient Descent. In Proc. of ICLR.
[38]
Qiheng Sun, Xiang Li, Jiayao Zhang, Li Xiong, Weiran Liu, Jinfei Liu, Zhan Qin, and Kui Ren. 2023. Shapleyfl: Robust federated learning based on shapley value. In Proc. of KDD.
[39]
Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. 2020. Data poisoning attacks against federated learning systems. In Proc. of ESORICS.
[40]
Fei Wang, Ethan Hugh, and Baochun Li. 2023. More than Enough is Too Much: Adaptive Defenses against Gradient Leakage in Production Federated Learning. In Proc. of INFOCOM.
[41]
Yansheng Wang, Yongxin Tong, Zimu Zhou, Ziyao Ren, Yi Xu, Guobin Wu, and Weifeng Lv. 2023. Fed-LTD: Towards cross-platform ride hailing via federated learning to dispatch. In Proc. of KDD.
[42]
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2023. FedAttack: Effective and covert poisoning attack on federated recommendation via hard sampling. In Proc. of KDD.
[43]
Xidong Wu, Zhengmian Hu, Jian Pei, and Heng Huang. 2023. Serverless federated auprc optimization for multi-party collaborative imbalanced data mining. In Proc. of KDD.
[44]
Zhaoxian Wu, Qing Ling, Tianyi Chen, and Georgios B Giannakis. 2020. Federated variance-reduced stochastic gradient descent with robustness to byzantine attacks. IEEE Transactions on Signal Processing, Vol. 68 (2020), 4583--4596.
[45]
Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).
[46]
Gang Yan, Hao Wang, Xu Yuan, and Jian Li. 2023. Criticalfl: A critical learning periods augmented client selection framework for efficient federated learning. In Proc. of KDD.
[47]
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. 2020. Rethinking bias-variance trade-off for generalization of neural networks. In Proc. of ICML.
[48]
Jingwei Yi, Fangzhao Wu, Bin Zhu, Jing Yao, Zhulin Tao, Guangzhong Sun, and Xing Xie. 2023. UA-FedRec: untargeted attack on federated news recommendation. In Proc. of KDD.
[49]
Dong Yin, Yudong Chen, and Ramchandran Kannan. 2019. Defending against saddle point attack in Byzantine-robust distributed learning. In Proc. of ICML.
[50]
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. 2018. Byzantine-robust distributed learning: Towards optimal statistical rates. In Proc. of ICML.
[51]
Chen Zhang, Boyang Zhou, Zhiqiang He, Zeyuan Liu, Yanjiao Chen, Wenyuan Xu, and Baochun Li. 2023. Oblivion: Poisoning Federated Learning by Inducing Catastrophic Forgetting. In Proc. of INFOCOM.
[52]
Xiaoli Zhang, Fengting Li, Zeyu Zhang, Qi Li, Cong Wang, and Jianping Wu. 2020. Enabling execution assurance of federated learning at untrusted participants. In Proc. of INFOCOM.
[53]
Zaixi Zhang, Xiaoyu Cao, Jinayuan Jia, and Neil Zhenqiang Gong. 2022. FLDetector: Defending Federated Learning Against Model Poisoning Attacks via Detecting Malicious Clients. In Proc. of KDD.
[54]
Yifeng Zheng, Shangqi Lai, Yi Liu, Xingliang Yuan, Xun Yi, and Cong Wang. 2023. Aggregation Service for Federated Learning: An Efficient, Secure, and More Resilient Realization. IEEE Transactions on Dependable and Secure Computing, Vol. 20, 2 (2023), 988--1001. https://doi.org/10.1109/TDSC.2022.3146448
[55]
Xiaoling Zhou, Ou Wu, Weiyao Zhu, and Ziyang Liang. 2022. Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure. In Proc. of ECML-PKDD.
Index Terms
- BadSampler: Harnessing the Power of Catastrophic Forgetting to Poison Byzantine-robust Federated Learning
Index terms have been assigned to the content through auto-classification.
Recommendations
Dynamic defense against byzantine poisoning attacks in federated learning
AbstractFederated learning, as a distributed learning that conducts the training on the local devices without accessing to the training data, is vulnerable to Byzantine poisoning adversarial attacks. We argue that the federated learning model ...
Highlights- We identify Byzantine attacks as a real problem of Federated Learning.
- We ...
Survey on federated learning threats: Concepts, taxonomy on attacks and defences, experimental study and challenges
AbstractFederated learning is a machine learning paradigm that emerges as a solution to the privacy-preservation demands in artificial intelligence. As machine learning, federated learning is threatened by adversarial attacks against the ...
Highlights- We claim that adversarial attacks are a significant challenge in federated learning.
AFLGuard: Byzantine-robust Asynchronous Federated Learning
ACSAC '22: Proceedings of the 38th Annual Computer Security Applications ConferenceFederated learning (FL) is an emerging machine learning paradigm, in which clients jointly learn a model with the help of a cloud server. A fundamental challenge of FL is that the clients are often heterogeneous, e.g., they have different computing ...
Comments
Information & Contributors
Information
Published In
August 2024
6901 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- City University of Hong Kong
- Hong Kong Research Grants Council
Conference
KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 25 - 29, 2024
Barcelona, Spain
Acceptance Rates
Overall Acceptance Rate 1,133 of 8,635 submissions, 13%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 81Total Downloads
- Downloads (Last 12 months)81
- Downloads (Last 6 weeks)81
Reflects downloads up to 06 Oct 2024
Other Metrics
Citations
Cited By
View all- Wu NLin XLu JZhang FChen WTang JXiao J(2024)Byzantine-Robust Multimodal Federated Learning Framework for Intelligent Connected VehicleElectronics10.3390/electronics1318363513:18(3635)Online publication date: 12-Sep-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in