Heterogeneous decentralised machine unlearning with seed model distillation
Pages 608 - 619
Abstract
As some recent information security legislation endowed users with unconditional rights to be forgotten by any trained machine learning model, personalised IoT service providers have to put unlearning functionality into their consideration. The most straightforward method to unlearn users' contribution is to retrain the model from the initial state, which is not realistic in high throughput applications with frequent unlearning requests. Though some machine unlearning frameworks have been proposed to speed up the retraining process, they fail to match decentralised learning scenarios. A decentralised unlearning framework called heterogeneous decentralised unlearning framework with seed (HDUS) is designed, which uses distilled seed models to construct erasable ensembles for all clients. Moreover, the framework is compatible with heterogeneous on‐device models, representing stronger scalability in real‐world applications. Extensive experiments on three real‐world datasets show that our HDUS achieves state‐of‐the‐art performance.
References
[1]
Imran, M., et al.: ReFRS: resource‐efficient federated recommender system for dynamic and diversified user preferences. ACM Trans. Inf. Syst. 41(3), 1–30 (2022)
[2]
Ye, G., et al.: Personalized on‐device e‐health analytics with decentralized block coordinate descent. IEEE J. Biomed. Health. Inform 26(6), 2778–2786 (2022)
[3]
Asare, K.O., et al.: Predicting depression from smartphone behavioral markers using machine learning methods, hyperparameter optimization, and feature importance analysis: exploratory study. JMIR mHealth and uHealth 9(7), e26540 (2021). https://doi.org/10.2196/26540
[4]
Harirchian, E., et al.: Ml‐ehsapp: a prototype for machine learning‐based earthquake hazard safety assessment of structures by using a smartphone app. Eur. J. Environ. Civ. Eng. 26(11), 5279–5299 (2022)
[5]
Chae, S.H., et al.: Development and clinical evaluation of a web‐based upper limb home rehabilitation system using a smartwatch and machine learning model for chronic stroke survivors: prospective comparative study. JMIR mHealth and uHealth 8(7), e17216 (2020)
[6]
Yang, Q., et al.: Federated learning. Synth. Lect. Artif. Intell. Mach. Learn 13(3), 1–207 (2019)
[7]
Boyd, S., et al.: Randomized gossip algorithms. IEEE Trans. Inf. Theor. 52(6), 2508–2530 (2006). https://doi.org/10.1109/TIT.2006.874516
[8]
Hsieh, K., et al.: The non‐iid data quagmire of decentralized machine learning. In: International Conference on Machine Learning, pp. 4387–4398. PMLR (2020)
[9]
Kasyap, H., Tripathy, S.: Privacy‐preserving decentralized learning framework for healthcare system. ACM Trans. Multimed Comput. Commun. Appl 17(2s), 1–24 (2021)
[10]
Kempe, D., Dobra, A., Gehrke, J.: Gossip‐based computation of aggregate information. In: 44th Annual IEEE Symposium on Foundations of Computer Science, 2003, pp. 482–491. IEEE (2003). Proceedings
[11]
Bistritz, I., Mann, A., Bambos, N.: Distributed distillation for on‐device learning. Adv. Neural Inf. Process. Syst. 33 (2020)
[12]
Long, J., et al.: Decentralized collaborative learning framework for next poi recommendation. ACM Trans. Inf. Syst. 41(3), 1–25 (2023)
[13]
Qu, L., et al.: Semi‐decentralized federated ego graph learning for recommendation. In: Proceedings of the ACM Web Conference 2023, pp. 339–348 (2023)
[14]
Long, J., et al.: Model‐agnostic decentralized collaborative learning for on‐device poi recommendation. In: Proceedings of the 46th International ACM SIGIR Conference on Research and D n n j8evelopment in Information Retrieval, pp. 423–432 (2023)
[15]
Chen, T., et al.: Learning elastic embeddings for customizing on‐device recommenders. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 138–147 (2021)
[16]
Cai, H., et al.: Once‐for‐all: train one network and specialize it for efficient deployment. ICLR (2020)
[17]
Nguyen, Q.V.H., et al.: Argument discovery via crowdsourcing. VLDB J. 26, 511–535 (2017)
[18]
Hung, N.Q.V., et al.: Computing crowd consensus with partial agreement. IEEE Trans. Knowl. Data Eng. 30(1), 1–14 (2017)
[19]
Seo, H., et al.: 16 federated knowledge distillation. Machine Learning and Wireless Communications, 457 (2022)
[20]
Yuan, L., et al.: Revisiting knowledge distillation via label smoothing regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903–3911 (2020)
[21]
Zhang, H., et al.: Adversarial co‐distillation learning for image recognition. Pattern Recogn. 111, 107659 (2021)
[22]
Voigt, P., Von dem Bussche, A.: The Eu General Data Protection Regulation (Gdpr). A Practical Guide, 1st Ed, vol. 10, pp. 10–5555. Springer International Publishing, Cham (2017).3152676
[23]
Pardau, S.L.: The California consumer privacy act: towards a european‐style privacy regime in the United States. J. Tech. L. Pol’y. 23, 68 (2018)
[24]
Shastri, S., Wasserman, M., Chidambaram, V.: The seven sins of {Personal‐Data} processing systems under {GDPR. In: 11th USENIX Workshop on Hot Topics in Cloud Computing. HotCloud 19 (2019)
[25]
Bhagoji, A.N., et al.: Analyzing federated learning through an adversarial lens. In: International Conference on Machine Learning, pp. 634–643. PMLR (2019)
[26]
Bourtoule, L., et al.: Machine unlearning. In: 2021 IEEE Symposium on Security and Privacy (SP), pp. 141–159. IEEE (2021)
[27]
Wu, C., Zhu, S., Mitra, P.: Federated unlearning with knowledge distillation. arXiv preprint (2022). arXiv:2201.09441
[28]
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT press (2016)
[29]
Foerster, J.N., et al.: Input switched affine networks: an rnn architecture designed for interpretability. In: International Conference on Machine Learning, pp. 1136–1145. PMLR (2017)
[30]
Aljundi, R., et al.: Memory aware synapses: learning what (not) to forget. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 139–154 (2018)
[31]
Abadi, M., et al.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318 (2016)
[32]
Dwork, C., et al.: The algorithmic foundations of differential privacy. Found. Trends® Theor. Comput. Sci. 9(3–4), 211–407 (2014)
[33]
Lee, J., Kifer, D.: Concentrated differentially private gradient descent with adaptive per‐iteration privacy budget. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1656–1665 (2018)
[34]
Ghorbani, A., Zou, J.: Data shapley: equitable valuation of data for machine learning. In: International Conference on Machine Learning, pp. 2242–2251. PMLR (2019)
[35]
Ene, A., Nguyen, H., Végh, L.A.: Decomposable submodular function minimization: discrete and continuous. Adv. Neural Inf. Process. Syst. 30 (2017)
[36]
Cao, Y., Yang, J.: Towards making systems forget with machine unlearning. In: 2015 IEEE Symposium on Security and Privacy, pp. 463–480. IEEE (2015)
[37]
Nguyen, Q.P., Low, B.K.H., Jaillet, P.: Variational bayesian unlearning. Adv. Neural Inf. Process. Syst. 33, 16025–16036 (2020)
[38]
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. USA 114(13), 3521–3526 (2017)
[39]
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint (2015). arXiv:1503.02531
[40]
Thudi, A., et al.: Unrolling sgd: understanding factors influencing machine unlearning. In: 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 303–319. IEEE (2022)
[41]
Lai, W.J., et al.: Date: a decentralized, anonymous, and transparent e‐voting system. In: 2018 1st IEEE International Conference on Hot Information‐Centric Networking (HotICN), pp. 24–29. IEEE (2018)
[42]
Nedić, A., Ozdaglar, A.: Convergence rate for consensus with delays. J. Global Optim. 47, 437–456 (2010)
[43]
Forero, P.A., Cano, A., Giannakis, G.B.: Consensus‐based distributed linear support vector machines. In: Proceedings of the 9th ACM/IEEE International Conference on Information Processing in Sensor Networks, pp. 35–46 (2010)
[44]
Iftikhar, S., et al.: Ai‐based fog and edge computing: a systematic review, taxonomy and future directions. Internet of Things 21, 100674 (2022)
[45]
Goldreich, O.: Secure multi‐party computation. Manuscript. Preliminary version 78(110) (1998)
[46]
Gentry, C.: Fully homomorphic encryption using ideal lattices. In: Proceedings of the Forty‐First Annual ACM Symposium on Theory of Computing, pp. 169–178 (2009)
[47]
Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms. CRC press (2012)
[48]
Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)
[49]
Schapire, R.E.: The strength of weak learnability. Mach. Learn. 5, 197–227 (1990)
[50]
Wolpert, D.H.: Stacked generalization. Neural Network. 5(2), 241–259 (1992)
[51]
Ye, G., et al.: Heterogeneous collaborative learning for personalized healthcare analytics via messenger distillation. IEEE J. Biomed. Health. Inform, 1–10 (2023). https://doi.org/10.1109/JBHI.2023.3247463
[52]
Dietterich, T.G.: Ensemble methods in machine learning. In: Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Italy, June 21–23, 2000 Proceedings, vol. 1, pp. 1–15. Springer (2000)
[53]
Rokach, L.: Ensemble‐based classifiers. Artif. Intell. Rev. 33, 1–39 (2010)
[54]
Demertzis, K., et al.: Federated auto‐meta‐ensemble learning framework for ai‐enabled military operations. Electronics 12(2), 430 (2023)
[55]
Yu, Y., et al.: Decentralized ensemble learning based on sample exchange among multiple agents. In: Proceedings of the 2019 ACM International Symposium on Blockchain and Secure Critical Infrastructure, pp. 57–66 (2019)
[56]
Papernot, N., et al.: Distillation as a defense to adversarial perturbations against deep neural networks. In: 2016 IEEE Symposium on Security and Privacy (SP), pp. 582–597. IEEE (2016)
[57]
Lian, X., et al.: Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Adv. Neural Inf. Process. Syst. 30 (2017)
[58]
McMahan, B., et al.: Communication‐efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)
[59]
Koloskova, A., et al.: A unified theory of decentralized sgd with changing topology and local updates. In: International Conference on Machine Learning, pp. 5381–5393. PMLR (2020)
[60]
LeCun, Y.: The Mnist Database of Handwritten Digits (1998). http://yann.lecun.com/exdb/mnist/
[61]
Xiao, H., Rasul, K., Vollgraf, R.: Fashion‐mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint (2017). arXiv:1708.07747
[62]
Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. Master’s thesis, University of Tront (2009)
[63]
He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
[64]
Howard, A., et al.: Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Index Terms
- Heterogeneous decentralised machine unlearning with seed model distillation
Index terms have been assigned to the content through auto-classification.
Recommendations
Hierarchical Machine Unlearning
Learning and Intelligent OptimizationAbstractIn recent years, deep neural networks have enjoyed tremendous success in industry and academia, especially for their applications in visual recognition and natural language processing. While large-scale deep models bring incredible performance, ...
BFU: Bayesian Federated Unlearning with Parameter Self-Sharing
ASIA CCS '23: Proceedings of the 2023 ACM Asia Conference on Computer and Communications SecurityAs the right to be forgotten has been legislated worldwide, many studies attempt to design machine unlearning mechanisms to enable data erasure from a trained model. Existing machine unlearning studies focus on centralized learning, where the server can ...
Machine Learning: The State of the Art
The two fundamental problems in machine learning (ML) are statistical analysis and algorithm design. The former tells us the principles of the mathematical models that we establish from the observation data. The latter defines the conditions on which ...
Comments
Information & Contributors
Information
Published In
© 2024 The Authors. CAAI Transactions on Intelligence Technology published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology and Chongqing University of Technology.
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
Publisher
John Wiley & Sons, Inc.
United States
Publication History
Published: 17 January 2024
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in