research-article

Deep Reinforcement Scheduling for Mobile Crowdsensing in Fog Computing

Authors:

Mianxiong DongAuthors Info & Claims

ACM Transactions on Internet Technology (TOIT), Volume 19, Issue 2

Article No.: 21, Pages 1 - 18

https://doi.org/10.1145/3234463

Published: 16 April 2019 Publication History

Abstract

Mobile crowdsensing becomes a promising technology for the emerging Internet of Things (IoT) applications in smart environments. Fog computing is enabling a new breed of IoT services, which is also a new opportunity for mobile crowdsensing. Thus, in this article, we introduce a framework enabling mobile crowdsensing in fog environments with a hierarchical scheduling strategy. We first introduce the crowdsensing framework that has a hierarchical structure to organize different resources. Since different positions and performance of fog nodes influence the quality of service (QoS) of IoT applications, we formulate a scheduling problem in the hierarchical fog structure and solve it by using a deep reinforcement learning–based strategy. From extensive simulation results, our solution outperforms other scheduling solutions for mobile crowdsensing in the given fog computing environment.

References

[1]

Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan, Nelson Huang, and Amin Vahdat. 2010. Hedera: Dynamic flow scheduling for data center networks. In Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation (NSDI’10). USENIX Association, 19--19. http://dl.acm.org/citation.cfm?id=1855711.1855730.

Digital Library

[2]

M. A. Alsheikh, D. Niyato, S. Lin, H. P. Tan, and Z. Han. 2016. Mobile big data analytics using deep learning and apache spark. IEEE Network 30, 3 (May 2016), 22--29.

[3]

S. Arai, K. Sycara, and T. R. Payne. 2000. Multi-agent reinforcement learning for planning and scheduling multiple goals. In Proceedings of the 4th International Conference on MultiAgent Systems. 359--360.

Digital Library

[4]

R. Atallah, C. Assi, and M. Khabbaz. 2017. Deep reinforcement learning-based scheduling for roadside communication networks. In Proceedings of the 2017 15th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt’17). 1--8.

[5]

S. Basudan, X. Lin, and K. Sankaranarayanan. 2017. A privacy-preserving vehicular crowdsensing-based road surface condition monitoring system using fog computing. IEEE Internet of Things Journal 4, 3 (June 2017), 772--782.

[6]

Richard Bellman. 2013. Dynamic Programming. Courier Corporation.

[7]

K. Bessai, S. Youcef, A. Oulamara, C. Godart, and S. Nurcan. 2012. Bi-criteria workflow tasks allocation and scheduling in cloud computing environments. In Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing. 638--645.

Digital Library

[8]

L. F. Bittencourt, J. Diaz-Montes, R. Buyya, O. F. Rana, and M. Parashar. 2017. Mobility-aware application scheduling in fog computing. IEEE Cloud Computing 4, 2 (March 2017), 26--35.

[9]

Flavio Bonomi, Rodolfo Milito, Preethi Natarajan, and Jiang Zhu. 2014. Fog Computing: A Platform for Internet of Things and Analytics. Springer International Publishing, Cham, 169--186.

[10]

Flavio Bonomi, Rodolfo Milito, Jiang Zhu, and Sateesh Addepalli. 2012. Fog computing and its role in the internet of things. In Proceedings of the 1st Edition of the MCC Workshop on Mobile Cloud Computing (MCC’12). ACM, New York, 13--16.

Digital Library

[11]

D. Bruneo, S. Distefano, F. Longo, G. Merlino, A. Puliafito, V. D’Amico, M. Sapienza, and G. Torrisi. 2016. Stack4Things as a fog computing platform for smart city applications. In Proceedings of the 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). 848--853.

[12]

Shin-Ming Cheng, Phone Lin, Di-Wei Huang, and Shun-Ren Yang. 2006. A study on distributed/centralized scheduling for wireless mesh network. In Proceedings of the 2006 International Conference on Wireless Communications and Mobile Computing (IWCMC’06). ACM, New York, 599--604.

Digital Library

[13]

Yohan Chon, Nicholas D. Lane, Fan Li, Hojung Cha, and Feng Zhao. 2012. Automatically characterizing places with opportunistic crowdsensing using smartphones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp’12). ACM, New York, 481--490.

Digital Library

[14]

Salvatore Distefano, Francesco Longo, and Marco Scarpa. 2015. QoS assessment of mobile crowdsensing services. Journal of Grid Computing 13, 4 (Dec. 2015), 629--650.

Digital Library

[15]

Nathan Eagle and Alex (Sandy) Pentland. 2005. CRAWDAD dataset mit/reality (v. 2005-07-01). Downloaded from https://crawdad.org/mit/reality/20050701.

[16]

Eyal Even-Dar and Yishay Mansour. 2001. Learning rates for Q-Learning. In Computational Learning Theory, David Helmbold and Bob Williamson (Eds.). Springer, Berlin, 589--604.

Digital Library

[17]

R. K. Ganti, F. Ye, and H. Lei. 2011. Mobile crowdsensing: Current state and future challenges. IEEE Communications Magazine 49, 11 (Nov. 2011), 32--39.

[18]

Robert Glaubius, Terry Tidwell, Christopher Gill, and William D. Smart. 2010. Real-time scheduling via reinforcement learning. In Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI’ 10). AUAI Press, Arlington, Virginia, 201--209. http://dl.acm.org/citation.cfm?id=3023549.3023573

Digital Library

[19]

D. Hoang and T. D. Dang. 2017. FBRC: Optimization of task scheduling in fog-based region and cloud. In 2017 IEEE Trustcom/BigDataSE/ICESS. 1109--1114.

[20]

Kirak Hong, David Lillethun, Umakishore Ramachandran, Beate Ottenwälder, and Boris Koldehofe. 2013. Mobile fog: A programming model for large-scale applications on the internet of things. In Proceedings of the S2nd ACM SIGCOMM Workshop on Mobile Cloud Computing (MCC’13). ACM, New York, 15--20.

Digital Library

[21]

H. Hu, Y. Wen, T. S. Chua, J. Huang, W. Zhu, and X. Li. 2016. Joint content replication and request routing for social video distribution over cloud CDN: A community clustering method. IEEE Transactions on Circuits and Systems for Video Technology 26, 7 (July 2016), 1320--1333.

[22]

H. Hu, Y. Wen, and D. Niyato. 2017. Public cloud storage-assisted mobile social video sharing: A supermodular game approach. IEEE Journal on Selected Areas in Communications 35, 3 (March 2017), 545--556.

Digital Library

[23]

H. Hu, Y. Wen, and D. Niyato. 2017. Spectrum allocation and bitrate adjustment for mobile social video sharing: Potential game with online QoS learning approach. IEEE Journal on Selected Areas in Communications 35, 4 (April 2017), 935--948.

[24]

S. Kosta, A. Aucinas, Pan Hui, R. Mortier, and Xinwen Zhang. 2012. ThinkAir: Dynamic resource allocation and parallel execution in the cloud for mobile code offloading. In 2012 Proceedings IEEE INFOCOM. 945--953.

[25]

H. Li, K. Ota, and M. Dong. 2018. Learning IoT in edge: Deep learning for the internet of things with edge computing. IEEE Network 32, 1 (Jan. 2018), 96--101.

[26]

H. Li, K. Ota, M. Dong, and M. Guo. 2017. Mobile crowdsensing in software defined opportunistic networks. IEEE Communications Magazine 55, 6 (2017), 140--145.

[27]

H. Li, K. Ota, M. Dong, A. Vasilakos, and K. Nagano. 2017. Multimedia processing pricing strategy in GPU-accelerated cloud computing. IEEE Transactions on Cloud Computing PP, 99 (2017), 1--1.

[28]

L. Li, K. Ota, M. Dong, and W. Borjigin. 2017. Eyes in the dark: Distributed scene understanding for disaster management. IEEE Transactions on Parallel and Distributed Systems 28, 12 (Dec. 2017), 3458--3471.

[29]

T. Li, Y. Liu, L. Gao, and A. Liu. 2017. A cooperative-based model for smart-sensing tasks in fog computing. IEEE Access 5 (2017), 21296--21311.

[30]

Yuanyuan Li, Zhiyang Li, Mianxiong Dong, Wenyu Qu, Changqing Ji, and Junfeng Wu. 2015. Efficient subspace skyline query based on user preference using MapReduce. Ad Hoc Networks 35, Supplement C (2015), 105--115. Special Issue on Big Data Inspired Data Sensing, Processing and Networking Technologies.

Digital Library

[31]

Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. 2016. Resource management with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks (HotNets’16). ACM, New York, 50--56.

Digital Library

[32]

Giovanni Merlino, Stamatis Arkoulis, Salvatore Distefano, Chrysa Papagianni, Antonio Puliafito, and Symeon Papavassiliou. 2016. Mobile crowdsensing as a service: A platform for applications on top of sensing clouds. Future Generation Computer Systems 56, Supplement C (2016), 623--639.

Digital Library

[33]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. 2013. Playing atari with deep reinforcement learning. CoRR abs/1312.5602 (2013). arxiv:1312.5602 http://arxiv.org/abs/1312.5602

[34]

M. Mohammadi, A. Al-Fuqaha, M. Guizani, and J. S. Oh. 2018. Semisupervised deep reinforcement learning in support of IoT and smart city services. IEEE Internet of Things Journal 5, 2 (April 2018), 624--635.

[35]

Xuan-Qui Pham and Eui-Nam Huh. 2016. Towards task scheduling in a cloud-fog computing system. In Proceedings of the 2016 18th Asia-Pacific Network Operations and Management Symposium (APNOMS’16). 1--4.

[36]

Mahadev Satyanarayanan. 2010. Mobile computing: The next decade. In Proceedings of the 1st ACM Workshop on Mobile Cloud Computing 8 Services: Social Networks and Beyond (MCS’10). ACM, New York, Article 5, 6 pages.

Digital Library

[37]

M. Satyanarayanan, P. Simoens, Y. Xiao, P. Pillai, Z. Chen, K. Ha, W. Hu, and B. Amos. 2015. Edge analytics in the internet of things. IEEE Pervasive Computing 14, 2 (Apr. 2015), 24--31.

Digital Library

[38]

S. K. Sharma and X. Wang. 2017. Live data analytics with collaborative edge and cloud processing in wireless IoT networks. IEEE Access 5 (2017), 4621--4635.

[39]

W. Sherchan, P. P. Jayaraman, S. Krishnaswamy, A. Zaslavsky, S. Loke, and A. Sinha. 2012. Using on-the-move mining for mobile crowdsensing. In Proceedings of the 2012 IEEE 13th International Conference on Mobile Data Management. 115--124.

Digital Library

[40]

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. 2017. Mastering the game of Go without human knowledge. Nature 550 (Oct. 2017), 354.

[41]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014). arxiv:1409.1556 http://arxiv.org/abs/1409.1556

[42]

Karolj Skala, Davor Davidovic, Enis Afgan, Ivan Sovic, and Zorislav Sojat. 2015. Scalable distributed computing hierarchy: Cloud, fog and dew computing. Open Journal of Cloud Computing (OJCC) 2, 1 (2015), 16--24.

[43]

M. Wang, Y. Cui, X. Wang, S. Xiao, and J. Jiang. 2018. Machine learning for networking: Workflow, advances and opportunities. IEEE Network 32, 2 (March 2018), 92--99.

[44]

Christopher J. C. H. Watkins and Peter Dayan. 1992. Q-learning. Machine Learning 8, 3 (May 1992), 279--292.

Digital Library

[45]

L. Xiao, Y. Li, G. Han, H. Dai, and H. V. Poor. 2018. A secure mobile crowdsensing game with deep reinforcement learning. IEEE Transactions on Information Forensics and Security 13, 1 (Jan. 2018), 35--47.

[46]

L. Xiao, X. Wan, C. Dai, X. Du, X. Chen, and M. Guizani. 2018. Security in mobile edge caching with reinforcement learning. ArXiv e-prints (Jan. 2018). arxiv:cs.CR/1801.05915

[47]

Yu Xiao, Pieter Simoens, Padmanabhan Pillai, Kiryong Ha, and Mahadev Satyanarayanan. 2013. Lowering the barriers to large-scale mobile crowdsensing. In Proceedings of the 14th Workshop on Mobile Computing Systems and Applications (HotMobile’13). ACM, New York, Article 9, 6 pages.

Digital Library

[48]

Shanhe Yi, Cheng Li, and Qun Li. 2015. A survey of fog computing: Concepts, applications and issues. In Proceedings of the 2015 Workshop on Mobile Big Data (Mobidata’15). ACM, New York, 37--42.

Digital Library

[49]

Wei Zhang and Thomas G. Dietterich. 1995. A reinforcement learning approach to job-shop scheduling. In Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2 (IJCAI’95). Morgan Kaufmann Publishers Inc., San Francisco, CA, 1114--1120. http://dl.acm.org/citation.cfm?id=1643031.1643044

Digital Library

[50]

D. Zhao, X. Y. Li, and H. Ma. 2014. How to crowdsource tasks truthfully without sacrificing utility: Online incentive mechanisms with budget constraint. In Proceedings of the IEEE Conference on Computer Communications (IEEE INFOCOM’14). 1213--1221.

[51]

Z. Zheng, L. X. Cai, M. Dong, X. Shen, and H. V. Poor. 2011. Constrained energy-aware AP placement with rate adaptation in WLAN mesh networks. In Proceedings of the 2011 IEEE Global Telecommunications Conference (GLOBECOM’11). 1--5.

Cited By

Xu JLi KChen YHuang J(2024)Optimal Task Scheduling and Resource Allocation for Self-Powered Sensors in Internet of Things: An Energy Efficient ApproachIEEE Transactions on Network and Service Management10.1109/TNSM.2024.342025421:4(4410-4420)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1109/TNSM.2024.3420254
Proietti Mattia GBeraldi R(2024)Online Decentralized Scheduling in Fog Computing for Smart Cities Based on Reinforcement LearningIEEE Transactions on Cognitive Communications and Networking10.1109/TCCN.2024.337821910:4(1551-1565)Online publication date: Aug-2024
https://doi.org/10.1109/TCCN.2024.3378219
Peng XSong SZhang XDong MOta K(2024)Task Offloading for IoAV Under Extreme Weather Conditions Using Dynamic Price Driven Double Broad Reinforcement LearningIEEE Internet of Things Journal10.1109/JIOT.2024.336011011:10(17021-17033)Online publication date: 15-May-2024
https://doi.org/10.1109/JIOT.2024.3360110
Show More Cited By

Index Terms

Deep Reinforcement Scheduling for Mobile Crowdsensing in Fog Computing
1. Networks

Recommendations

Fog Computing: Issues and Challenges in Security and Forensics
COMPSAC '15: Proceedings of the 2015 IEEE 39th Annual Computer Software and Applications Conference - Volume 03

Although Fog Computing is defined as the extension of the Cloud Computing paradigm, its distinctive characteristics in the location sensitivity, wireless connectivity, and geographical accessibility create new security and forensics issues and ...
A Pattern for Fog Computing
VikingPLoP '16: Proceedings of the 10th Travelling Conference on Pattern Languages of Programs

Fog Computing is a new variety of the cloud computing paradigm that brings virtualized cloud services to the edge of the network to control the devices in the IoT. We present a pattern for fog computing which describes its architecture, including its ...
From Cloud Computing to Fog Computing: Platforms for the Internet of Things (IoT)

This article describes how in recent years, Cloud Computing has emerged as a fundamental computing paradigm that has significantly changed the approach of enterprises as well as end users towards implementation of Internet technology. The key ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet Technology

ACM Transactions on Internet Technology Volume 19, Issue 2

Special Issue on Fog, Edge, and Cloud Integration

May 2019

288 pages

ISSN:1533-5399

EISSN:1557-6051

DOI:10.1145/3322882

Editor:
Ling Liu
Georgia Institute of Technology, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 April 2019

Accepted: 01 June 2018

Revised: 01 May 2018

Received: 01 December 2017

Published in TOIT Volume 19, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Japan Society for the Promotion of Science
KDDI Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

86
Total Citations
View Citations
1,079
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)5

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu JLi KChen YHuang J(2024)Optimal Task Scheduling and Resource Allocation for Self-Powered Sensors in Internet of Things: An Energy Efficient ApproachIEEE Transactions on Network and Service Management10.1109/TNSM.2024.342025421:4(4410-4420)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1109/TNSM.2024.3420254
Proietti Mattia GBeraldi R(2024)Online Decentralized Scheduling in Fog Computing for Smart Cities Based on Reinforcement LearningIEEE Transactions on Cognitive Communications and Networking10.1109/TCCN.2024.337821910:4(1551-1565)Online publication date: Aug-2024
https://doi.org/10.1109/TCCN.2024.3378219
Peng XSong SZhang XDong MOta K(2024)Task Offloading for IoAV Under Extreme Weather Conditions Using Dynamic Price Driven Double Broad Reinforcement LearningIEEE Internet of Things Journal10.1109/JIOT.2024.336011011:10(17021-17033)Online publication date: 15-May-2024
https://doi.org/10.1109/JIOT.2024.3360110
Ding TLiu LZhu YCui LYan Z(2024)IoV environment exploring coordination: A federated learning approachDigital Communications and Networks10.1016/j.dcan.2022.07.00610:1(135-141)Online publication date: Feb-2024
https://doi.org/10.1016/j.dcan.2022.07.006
Li HYu JCao LZhang QSong ZHou S(2024)Multi-agent reinforcement learning based computation offloading and resource allocation for LEO Satellite edge computing networksComputer Communications10.1016/j.comcom.2024.05.008222:C(268-276)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.comcom.2024.05.008
Kaur NMittal ALilhore USimaiya SDalal SSharma Y(2024)An adaptive mobility-aware secure handover and scheduling protocol for Earth Observation (EO) communication using fog computingEarth Science Informatics10.1007/s12145-024-01291-w17:3(2429-2446)Online publication date: 28-Mar-2024
https://doi.org/10.1007/s12145-024-01291-w
Lera IGuerrero C(2024)Multi-objective application placement in fog computing using graph neural network-based reinforcement learningThe Journal of Supercomputing10.1007/s11227-024-06439-580:19(27073-27094)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s11227-024-06439-5
Keshavarz Haddadha PRezvani MMollaMotalebi MShankar A(2024)Machine learning methods for service placement: a systematic reviewArtificial Intelligence Review10.1007/s10462-023-10684-057:3Online publication date: 17-Feb-2024
https://doi.org/10.1007/s10462-023-10684-0
G. Harkut DThakar PMutneja L(2023)An Adaptive Task Scheduling in Fog ComputingSearch Algorithm - Essence of Optimization10.5772/intechopen.108785Online publication date: 1-Feb-2023
https://doi.org/10.5772/intechopen.108785
Kim HBen-Othman J(2023)BeneWinD: An Adaptive Benefit Win–Win Platform with Distributed Virtual Emotion FoundationElectronics10.3390/electronics1217354012:17(3540)Online publication date: 22-Aug-2023
https://doi.org/10.3390/electronics12173540
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents