Multi-Timescale Actor-Critic Learning for Computing Resource Management With Semi-Markov Renewal Process Mobility
Pages 452 - 461
Abstract
This paper studies artificial intelligence (AI) aided communication and computing resource allocation in a vehicular network that supports blockchain-enabled video streaming. Our study aims to improve the operating efficiency and to maximize the transcoding rewards for blockchain based vehicular networks. Our resource allocation policy considers the vehicular mobility, which is modelled with a highly-realistic Semi-Markov renewal process, as well as the real-time video service delay constraints. We propose a multi-timescale actor-critic-reinforcement learning framework to tackle these grand challenges. We also develop a prediction model for the vehicular mobility by using analysis and classical machine learning, which alleviates the heavy signaling and computation overheads due to the vehicular movement. A mobility-aware reward estimation for the large timescale model is then proposed to mitigate the complexity due to the large action space. Finally, numerical results are presented to illustrate the developed theoretical findings in this paper and the significant performance gains due to our proposed multi-timescale framework.
References
[1]
X. Jiang, F. R. Yu, T. Song, and V. C. M. Leung, “Resource allocation of video streaming over vehicular networks: A survey, some research issues and challenges,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 7, pp. 5955–5975, Jul. 2022.
[2]
P. Arthurs, L. Gillam, P. Krause, N. Wang, K. Halder, and A. Mouzakitis, “A taxonomy and survey of edge cloud computing for intelligent transportation systems and connected vehicles,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 7, pp. 6206–6221, Jul. 2022.
[3]
Q. Wang, L. T. Tan, R. Q. Hu, and Y. Qian, “Hierarchical energy-efficient mobile-edge computing in IoT networks,” IEEE Internet Things J., vol. 7, no. 12, pp. 11626–11639, Dec. 2020.
[4]
L. T. Tan, R. Q. Hu, and L. Hanzo, “Heterogeneous networks relying on full-duplex relays and mobility-aware probabilistic caching,” IEEE Trans. Commun., vol. 67, no. 7, pp. 5037–5052, Jul. 2019.
[5]
L. T. Tan, R. Q. Hu, and L. Hanzo, “Twin-timescale artificial intelligence aided mobility-aware edge caching and computing in vehicular networks,” IEEE Trans. Veh. Technol., vol. 68, no. 4, pp. 3086–3099, Apr. 2019.
[6]
K. Muhammad, A. Ullah, J. Lloret, J. D. Ser, and V. H. C. de Albuquerque, “Deep learning for safe autonomous driving: Current challenges and future directions,” IEEE Trans. Intell. Transp. Syst., vol. 22, no. 7, pp. 4316–4336, Jul. 2021.
[7]
L. T. Tan and R. Q. Hu, “Mobility-aware edge caching and computing in vehicle networks: A deep reinforcement learning,” IEEE Trans. Veh. Technol., vol. 67, no. 11, pp. 10190–10203, Nov. 2018.
[8]
M. S. Ali, M. Vecchio, M. Pincheira, K. Dolui, F. Antonelli, and M. H. Rehmani, “Applications of blockchains in the Internet of Things: A comprehensive survey,” IEEE Commun. Surveys Tuts., vol. 21, no. 2, pp. 1676–1717, 2nd Quart., 2019.
[9]
Transcodium: A Decentralized Peer-to-Peer Media Editing, Transcoding & Distribution Platform. Accessed: Aug. 12, 2023. [Online]. Available: https://www.allcryptowhitepapers.com/transcodium-whitepaper
[10]
M. Liu, Y. Teng, F. R. Yu, V. C. M. Leung, and M. Song, “A deep reinforcement learning-based transcoder selection framework for blockchain-enabled wireless D2D transcoding,” IEEE Trans. Commun., vol. 68, no. 6, pp. 3426–3439, Jun. 2020.
[11]
M. Liu, F. R. Yu, Y. Teng, V. C. M. Leung, and M. Song, “Distributed resource allocation in blockchain-based video streaming systems with mobile edge computing,” IEEE Trans. Wireless Commun., vol. 18, no. 1, pp. 695–708, Jan. 2019.
[12]
D. Liu, A. Alahmadi, J. Ni, X. Lin, and X. Shen, “Anonymous reputation system for IIoT-enabled retail marketing atop PoS blockchain,” IEEE Trans. Ind. Informat., vol. 15, no. 6, pp. 3527–3537, Jun. 2019.
[13]
A. Haydari and Y. Yilmaz, “Deep reinforcement learning for intelligent transportation systems: A survey,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 1, pp. 11–32, Jan. 2022.
[14]
Q. Wang, G. M. Garrity, J. M. Tiedje, and J. R. Cole, “Naïve Bayesian classifier for rapid assignment of rRNA sequences into the new bacterial taxonomy,” Appl. Environ. Microbiol., vol. 73, no. 16, pp. 5261–5267, Aug. 2007.
[15]
T. Le and S. Shetty, “Artificial intelligence-aided privacy preserving trustworthy computation and communication in 5G-based IoT networks,” Ad Hoc Netw., vol. 126, Mar. 2022, Art. no.
[16]
S. Bhatnagar, R. S. Sutton, M. Ghavamzadeh, and M. Lee, “Natural actor–critic algorithms,” Automatica, vol. 45, no. 11, pp. 2471–2482, 2009.
[17]
R. Wang, J. Zhang, S. H. Song, and K. B. Letaief, “Mobility-aware caching in D2D networks,” IEEE Trans. Wireless Commun., vol. 16, no. 8, pp. 5001–5015, Aug. 2017.
[18]
S. Mozaffari, O. Y. Al-Jarrah, M. Dianati, P. Jennings, and A. Mouzakitis, “Deep learning-based vehicle behavior prediction for autonomous driving applications: A review,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 1, pp. 33–47, Jan. 2022.
[19]
S. Aradi, “Survey of deep reinforcement learning for motion planning of autonomous vehicles,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 2, pp. 740–759, Feb. 2022.
[20]
B. R. Kiranet al., “Deep reinforcement learning for autonomous driving: A survey,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 6, pp. 4909–4926, Jun. 2022.
[21]
X. Jiang, F. R. Yu, T. Song, and V. C. M. Leung, “A survey on multi-access edge computing applied to video streaming: Some research issues and challenges,” IEEE Commun. Surveys Tuts., vol. 23, no. 2, pp. 871–903, 2nd Quart., 2021.
[22]
R. S. Sutton, D. Precup, and S. Singh, “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning,” Artif. Intell., vol. 112, nos. 1–2, pp. 181–211, Aug. 1999.
[23]
T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum, “Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation,” in Proc. NIPS, vol. 29, 2016, pp. 1–9.
[24]
H. Soo Chang, P. J. Fard, S. I. Marcus, and M. Shayman, “Multitime scale Markov decision processes,” IEEE Trans. Autom. Control, vol. 48, no. 6, pp. 976–987, Jun. 2003.
[25]
H. Farooq, A. Asghar, and A. Imran, “Mobility prediction-based autonomous proactive energy saving (AURORA) framework for emerging ultra-dense networks,” IEEE Trans. Green Commun. Netw., vol. 2, no. 4, pp. 958–971, Dec. 2018.
[26]
J.-K. Lee and J. C. Hou, “Modeling steady-state and transient behaviors of user mobility: Formulation, analysis, and application,” in Proc. ACM MobiHoc, 2006, pp. 85–96.
[27]
T. Le, M. Reisslein, and S. Shetty. Multi-Timescale Actor-Critic Learning for Computing Resource Management With Semi-Markov Renewal Process Mobility (Extended Version). Accessed: Aug. 12, 2023. [Online]. Available: https://www.dropbox.com/s/k4m8d4rs1oyqjtn/ABTCVNTechnicalreport.pdf?dl=0
[28]
Y. Lv, Y. Duan, W. Kang, Z. Li, and F.-Y. Wang, “Traffic flow prediction with big data: A deep learning approach,” IEEE Trans. Intell. Transp. Syst., vol. 16, no. 2, pp. 865–873, Apr. 2015.
[29]
I. Rhee, M. Shin, S. Hong, K. Lee, S. J. Kim, and S. Chong, “On the Levy-walk nature of human mobility,” IEEE/ACM Trans. Netw., vol. 19, no. 3, pp. 630–643, Jun. 2011.
[30]
Q. Yuan, I. Cardei, and J. Wu, “An efficient prediction-based routing in disruption-tolerant networks,” IEEE Trans. Parallel Distrib. Syst., vol. 23, no. 1, pp. 19–31, Jan. 2012.
Index Terms
- Multi-Timescale Actor-Critic Learning for Computing Resource Management With Semi-Markov Renewal Process Mobility
Index terms have been assigned to the content through auto-classification.
Recommendations
Markov-Renewal Programming. II: Infinite Return Models, Example
This paper is a continuation of a previous one which investigates programming over a Markov-renewal process---in which the intervals between transitions of a system from state i to state j are independent samples from a distribution that may depend upon ...
Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning
<P>A large class of problems of sequential decision making under uncertainty, of which the underlying probability structure is a Markov process, can be modeled as stochastic dynamic programs referred to, in general, as Markov decision problems or MDPs. ...
Actor-critic algorithms for hierarchical Markov decision processes
We consider the problem of control of hierarchical Markov decision processes and develop a simulation based two-timescale actor-critic algorithm in a general framework. We also develop certain approximation algorithms that require less computation and ...
Comments
Information & Contributors
Information
Published In
1558-0016 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 22 August 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 27 Jul 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in