Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

Reinforcement Learning for Real-Time Federated Learning for Resource-Constrained Edge Cluster

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

For performing various predictive analytics tasks for real-time mission-critical applications, Federated Learning (FL) have emerged as the go-to machine learning paradigm for its ability to leverage perform machine learning workloads on resource-constrained edge devices. For such FL applications working under stringent deadlines, the overall local training time needs to be minimized, which consists of the retrieval delay, i.e., the delay in fetching the data from the IoT devices to the FL clients as well as the time consumed in training the local models. Since the latter component is mostly uniform among the FL clients, we have to minimize the retrieval delay to reduce the local training time. To that end, we formulate the Client Assignment Problem (CAP) as an intelligent assignment of selected IoT devices to each FL client such that the FL client may retrieve training data from these IoT devices with minimal retrieval delay. CAP must perform assignments for each FL client considering its relative distances from each IoT device such that each FL client does not experience an arbitrarily large retrieval delay in fetching data from a remotely placed IoT device. We prove that CAP is NP-Hard, and as such, obtaining a polynomial time solution to CAP is infeasible. To deal with the challenges faced by such heuristics approaches, we propose Deep Reinforcement Learning-based algorithms to produce near-optimal solution to CAP. We demonstrate that our algorithms outperform the state of the art in reducing the local training time, while producing a near-optimal solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. We consider the retrieval delay as the delay in fetching the data at an FL client from the IoT device assigned to it.

  2. We denote client assignment as a possible connection of the IoT devices to each FL client in the cluster in a specific fashion.

  3. We use clients and FL clients interchangeably.

  4. https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/running-applications-in-containers.html

  5. FedAvg is sensitive to data heterogeneity, and in our case, this requires a more focused study on the FL aggregation algorithms, which is subject to future work.

  6. In future work, we consider the mobility of IoT devices and clients, and its effect on federated learning.

  7. We follow the lead of prior papers [60], which denotes the general utilization of computational resources observed while running an application or a task as the load The load experienced on an FL client while training a local model is a significant optimization parameter that must be considered when assigning IoT devices to FL clients. The load at an instant can not exceed the maximum capacity (maximum load) that the FL clients can handle.

  8. For example, \(\mathcal {T}=2\) implies that the maximum number of IoT devices that can be assigned to a client is 2.

  9. https://stable-baselines.readthedocs.io/en/master/guide/algos.html

  10. https://spinningup.openai.com/en/latest/algorithms/ppo.html

  11. Please refer to [73] for more information on PPO.

  12. https://gymnasium.farama.org/

  13. On request, we share the code.

  14. A tabu move is a forbidden move that cannot be included in the recent solution list obtained.

  15. https://www.scipopt.org/

  16. The number of experiments performed are 30.

  17. These plots are obtained from the tensorboard log files.

  18. https://www.cs.toronto.edu/~kriz/cifar.html

  19. https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html

References

  1. Liu, S., et al.: Edge computing for autonomous driving: Opportunities and challenges. Proc. IEEE 107, 1697–1716 (2019)

    Article  Google Scholar 

  2. Rajashekar, K., Paul, S., Karmakar, S., Sidhanta, S.: Topology aware cluster configuration for minimizing communication delay in edge computing 1310–1311 (2022)

  3. Spatharakis, D.: et al. A scalable edge computing architecture enabling smart offloading for location based services. Pervasive Mob. Comput. 67, 101217 (2020). https://www.sciencedirect.com/science/article/pii/S1574119220300778

  4. Zinonos, Z., Vassiliou, V., Ioannou, C., Koutroullos, M.: Dynamic topology control for wsns in critical environments 1–5 (2011)

  5. Xia, Q., Ye, W., Tao, Z., Wu, J., Li, Q.: A survey of federated learning for edge computing: Research problems and solutions. High-Conf. Comput. 1, 100008 (2021). https://www.sciencedirect.com/science/article/pii/S266729522100009X

  6. Nguyen, D.C., et al.: Federated learning meets blockchain in edge computing: Opportunities and challenges. IEEE Int. Things J. 8, 12806–12825 (2021)

    Article  Google Scholar 

  7. Perkin, T.M., Mini, S.: Assignment of iot nodes to edge computing devices in internet of things 528–532 (2019)

  8. Sprague, M.R., et al.: Asynchronous federated learning for geospatial applications 21–28 (2018)

  9. Yang, J., Zhou, Y., Wen, W., Zhou, J., Zhang, Q.: Asynchronous hierarchical federated learning based on bandwidth allocation and client scheduling. Appl. Sci. 13, 11134 (2023)

    Article  Google Scholar 

  10. Fan, Q., Ansari, N.: Application aware workload allocation for edge computing-based iot. IEEE Int. Things J. 5, 2146–2153 (2018)

    Article  Google Scholar 

  11. Saha, R., Misra, S., Deb, P.K.: Fogfl: Fog-assisted federated learning for resource-constrained iot devices. IEEE Int. Things J. 8, 8456–8463 (2021)

    Article  Google Scholar 

  12. Ji, Y., et al.: Client selection and bandwidth allocation for federated learning: An online optimization perspective 5075–5080 (2022)

  13. Sudharsan, B., Breslin, J.G., Ali, M.I.: Edge2train: A framework to train machine learning models (svms) on resource-constrained iot edge devices 1–8 (2020)

  14. Rajashekar, K., Paul, S., Karmakar, S., Sidhanta, S.: Minimizing data retrieval delay in edge computing 63–85 (2023)

  15. Nijimbere, D., Zhao, S., Gu, X., Esangbedo, M.O., Dominique, N.: Tabu search guided by reinforcement learning for the max-mean dispersion problem. J. Indust. Manag. Opt. 17, 3223–3246 (2021)

    Article  MathSciNet  Google Scholar 

  16. Rajashekar, K.: Reinforcement learning for minimizing communication delay in edge computing 1270–1271 (2022)

  17. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA (2018)

    Google Scholar 

  18. Xu, C., Qu, Y., Xiang, Y., Gao, L.: Asynchronous federated learning on heterogeneous devices: A survey. Comput. Sci. Rev. 50, 100595 (2023)

    Article  Google Scholar 

  19. Fan, B., Su, X., Tarkoma, S., Hui, P.: Behave differently when clustering: a semi-asynchronous federated learning approach for iot. ACM Trans. Sensor Netw. 20, 1–28 (2024)

    Article  Google Scholar 

  20. Lu, X., Liao, Y., Lio, P., Hui, P.: Privacy-preserving asynchronous federated learning mechanism for edge network computing. Ieee Access 8, 48970–48981 (2020)

    Article  Google Scholar 

  21. Fang, C., Shi, L., Shi, Y., Xu, J., Ding, X.: Synchronous federated learning latency optimization based on model splitting 495–506 (2022)

  22. Wen, J., et al.: A survey on federated learning: challenges and applications. Int. J. Mach. Learn. Cybernet. 14, 513–535 (2023)

    Article  Google Scholar 

  23. Xiao, Y., et al.: Time-sensitive learning for heterogeneous federated edge intelligence. IEEE Trans. Mob. Comput. 23, 1382–1400 (2024)

    Article  Google Scholar 

  24. Konečnỳ, J.: et al. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016)

  25. Li, T., et al.: Federated optimization in heterogeneous networks. Proc. Mach. Learn. Syst. 2, 429–50 (2020)

    Google Scholar 

  26. Kairouz, P.: et al. Advances and open problems in federated learning. Found. Trends. Mach. Learn. 14, 1–210 (2021)

  27. Yousefpour, A., Ishigaki, G., Jue, J.P.: Fog computing: Towards minimizing delay in the internet of things 17–24 (2017)

  28. Song, Y., Yau, S.S., Yu, R., Zhang, X., Xue, G.: An approach to qos-based task distribution in edge computing networks for iot applications. In: Proceedings - 2017 IEEE 1st International Conference on Edge Computing, EDGE 2017 32–39 (2017)

  29. Kherraf, N., Sharafeddine, S., Assi, C.M., Ghrayeb, A.: Latency and reliability-aware workload assignment in iot networks with mobile edge clouds. IEEE Trans. Netw. Serv. Manag. 16, 1435–1449 (2019)

    Article  Google Scholar 

  30. Sun, X., Ansari, N.: Latency aware workload offloading in the cloudlet network. IEEE Commun. Lett. 21, 1481–1484 (2017)

    Article  Google Scholar 

  31. Wei, Z., Jiang, H.: Optimal offloading in fog computing systems with non-orthogonal multiple access. IEEE Access 6, 49767–49778 (2018)

    Article  Google Scholar 

  32. Sheng, M., et al.: Delay-aware computation offloading in noma mec under differentiated uploading delay. IEEE Trans. Wireless Commun. 19, 2813–2826 (2020)

    Article  Google Scholar 

  33. Lin, K.C.-J., Wang, H.-C., Lai, Y.-C., Lin, Y.-D.: Communication and computation offloading for multi-rat mobile edge computing. IEEE Wireless Commun. 26, 180–186 (2019)

    Article  Google Scholar 

  34. Hua, H., et al.: Edge computing with artificial intelligence: A machine learning perspective. ACM Comput. Surv. 55, 1–35 (2023)

    Article  Google Scholar 

  35. Huang, L., Bi, S., Zhang, Y.-J.A.: Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks. IEEE Trans. Mob. Comput. 19, 2581–2593 (2020)

    Article  Google Scholar 

  36. Liu, H., Cao, G.: Deep reinforcement learning-based server selection for mobile edge computing. IEEE Trans. Veh. Technol. 70, 13351–13363 (2021)

    Article  Google Scholar 

  37. Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: Challenges, methods, and future directions. IEEE Signal Process. Mag. 37, 50–60 (2020)

    Google Scholar 

  38. Qiao, Z., et al.: Content-aware client selection for federated learning in wireless networks 49–54 (2022)

  39. Imteaj, A., Thakker, U., Wang, S., Li, J., Amini, M.H.: A survey on federated learning for resource-constrained iot devices. IEEE Int. Things J. 9, 1–24 (2022)

    Article  Google Scholar 

  40. Park, J., Han, D.-J., Choi, M., Moon, J., Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J. W., eds.: Sageflow: Robust federated learning against both stragglers and adversaries. (eds Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. & Vaughan, J. W.) Advances in Neural Information Processing Systems, Vol. 34, 840–851 (Curran Associates, Inc., 2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/076a8133735eb5d7552dc195b125a454-Paper.pdf

  41. Xu, Z., Yang, Z., Xiong, J., Yang, J., Chen, X.: Elfish: Resource-aware federated learning on heterogeneous edge devices. Ratio 2, r2 (2019)

    Google Scholar 

  42. Amadeo, M., Campolo, C., Molinaro, A., Ruggeri, G., Singh, G.: Mitigating the communication straggler effect in federated learning via named data networking. IEEE Commun. Mag. (2024)

  43. Feng, J., Liu, L., Pei, Q., Li, K.: Min-max cost optimization for efficient hierarchical federated learning in wireless edge networks. IEEE Trans. Parallel Distrib. Syst. 33, 2687–2700 (2022)

    Google Scholar 

  44. Liu, T., Zhou, H., Li, J., Shu, F., Han, Z.: Uplink and downlink decoupled 5g/b5g vehicular networks: A federated learning assisted client selection method. IEEE Trans. Veh. Technol. 72, 2280–2292 (2022)

    Article  Google Scholar 

  45. Zheng, S., Shen, C., Chen, X.: Design and analysis of uplink and downlink communications for federated learning. IEEE J. Sel. Areas Commun. 39, 2150–2167 (2021)

    Article  Google Scholar 

  46. Baccarelli, E., Scarpiniti, M., Momenzadeh, A., Ahrabi, S.S.: Afafed-asynchronous fair adaptive federated learning for iot stream applications. Comput. Commun. 195, 376–402 (2022)

    Article  Google Scholar 

  47. Khan, A., et al.: Adaptive filtering: issues, challenges, and best-fit solutions using particle swarm optimization variants. Sensors 23, 7710 (2023)

    Article  Google Scholar 

  48. Min, M., et al.: Learning-based computation offloading for iot devices with energy harvesting. IEEE Trans. Veh. Technol. 68, 1930–1941 (2019)

    Article  Google Scholar 

  49. WANG, L., WANG, W., LI, B.: Cmfl: Mitigating communication overhead for federated learning 954–964 (2019)

  50. Wang, S., et al.: Adaptive federated learning in resource constrained edge computing systems. IEEE J. Sel. Areas Commun. 37, 1205–1221 (2019)

    Article  Google Scholar 

  51. Ludwig, H., Baracaldo, N.: Federated learning: A comprehensive overview of methods and applications. Springer, NY (2022)

    Book  Google Scholar 

  52. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data 1273–1282 (2017)

  53. Dai, Y., Xu, D., Maharjan, S., Zhang, Y.: Joint computation offloading and user association in multi-task mobile edge computing. IEEE Trans. Veh. Technol. 67, 12313–12325 (2018)

    Article  Google Scholar 

  54. Yu, L., Albelaihi, R., Sun, X., Ansari, N., Devetsikiotis, M.: Jointly optimizing client selection and resource management in wireless federated learning for internet of things. IEEE Int. Things J. 9, 4385–4395 (2022)

    Article  Google Scholar 

  55. Yan, Z., et al.: Exploiting edge computing in internet of space things networks: Dynamic and static server placement 1–6 (2021)

  56. Schempp, P., Preuß, K., Tröger, M.: About the correlation between crude oil corrosiveness and results from corrosion monitoring in an oil refinery. Corrosion 72, 843–855 (2016)

    Article  Google Scholar 

  57. Barthélemy, J., Verstaevel, N., Forehead, H., Perez, P.: Edge-computing video analytics for real-time traffic monitoring in a smart city. Sensors19 (2019). https://www.mdpi.com/1424-8220/19/9/2048

  58. Bonomi, F., Milito, R., Zhu, J., Addepalli, S.: Fog computing and its role in the internet of things 13–16 (2012). https://doi.org/10.1145/2342509.2342513

  59. Zhang, X., Li, Y., Li, W., Guo, K., Shao, Y.: Personalized federated learning via variational bayesian inference 26293–26310 (2022)

  60. Martello, S., Toth, P.: The bottleneck generalized assignment problem. Eur. J. Oper. Res. 83, 621–638 (1995)

    Article  Google Scholar 

  61. Martello, S., Toth, P.: Knapsack problems: algorithms and computer implementations. John Wiley & Sons Inc., US (1990)

    Google Scholar 

  62. Mazzola, J., Neebe, A.: Bottleneck generalized assignment problems. Eng. Costs Prod. Econ. 14, 61–65 (1988)

    Article  Google Scholar 

  63. Khosravanian, R., Mansouri, V., Wood, D.A., Alipour, M.R.: A comparative study of several metaheuristic algorithms for optimizing complex 3-d well-path designs. J. Pet. Explor. Prod. Technol. 8, 1487–1503 (2018). https://doi.org/10.1007/s13202-018-0447-2

    Article  Google Scholar 

  64. Mazyavkina, N., Sviridov, S., Ivanov, S., Burnaev, E.: Reinforcement learning for combinatorial optimization: A survey. Comput. Oper. Res.134, 105400 (2021). https://www.sciencedirect.com/science/article/pii/S0305054821001660

  65. Sayed, A.H.: Fundamentals of adaptive filtering. John Wiley & Sons, US (2003)

    Google Scholar 

  66. Lin, H., Lu, K., Wang, Y.: Adaptive filtering algorithm based on reinforcement learning 5268–5272 (2024)

  67. Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)

    Article  Google Scholar 

  68. Liu, D., Kong, H., Luo, X., Liu, W., Subramaniam, R.: Bringing ai to edge: from deep learning’s perspective. Neurocomputing (2021)

  69. Pathan, S., Shrivastava, V.: Reinforcement learning for assignment problem with time constraints 2106, 02856 (2021)

  70. Ahsan, W., Yi, W., Liu, Y., Qin, Z., Nallanathan, A.: Reinforcement learning for user clustering in noma-enabled uplink iot 1–6 (2020)

  71. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992)

    Article  Google Scholar 

  72. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. CoRR:abs/1602.01783 (2016). URL arXiv:1602.01783

  73. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017). arxiv:1707.06347

  74. Towers, M., et al.: Gymnasium (2023). https://zenodo.org/record/8127025

  75. Brockman, G., et al.: Openai gym. arXiv preprint arXiv:1606.01540 (2016)

  76. Wikipedia. Nearest neighbor search — Wikipedia, the free encyclopedia (2022). http://en.wikipedia.org/w/index.php?title=Nearest_neighbor_20search &oldid=1068801798. [Online; accessed 31-January-2022]

  77. Perron, L., Furnon, V.: Or-tools. https://developers.google.com/optimization/

  78. Design optimization. http://apmonitor.com/me575/index.php/Main/MiniMax

  79. Yang, L., et al.: Multi-uav-enabled load-balance mobile-edge computing for iot networks. IEEE Int. Things J. 7, 6898–6908 (2020)

    Article  Google Scholar 

Download references

Funding

No funding was received for conducting this work.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Bhilai, Durg, 491001, Chhattisgarh, India

    Kolichala Rajashekar & Souradyuti Paul

  2. Department of Computer Science and Engineering, Indian Institute of Technology Guwahati, Guwahati, 781001, Assam, India

    Sushanta Karmakar

  3. Department of Industrial Systems and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, 721302, West Bengal, India

    Subhajit Sidhanta

Authors
  1. Kolichala Rajashekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Souradyuti Paul
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sushanta Karmakar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Subhajit Sidhanta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kolichala Rajashekar.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence this work

Ethical Approval

This research work does not involve human participants and/or animals.

Consent to Participate

This research work has not been carried out on human participants and/or animals.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rajashekar, K., Paul, S., Karmakar, S. et al. Reinforcement Learning for Real-Time Federated Learning for Resource-Constrained Edge Cluster. J Netw Syst Manage 32, 94 (2024). https://doi.org/10.1007/s10922-024-09857-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10922-024-09857-1

Keywords

Search

Navigation