Cooperative Multi-Agent Deep Reinforcement Learning for Computation Offloading in Digital Twin Satellite Edge Networks
Authors: 
Zhe Ji, Sheng Wu, and Chunxiao JiangAuthors Info & Claims
Zhe Ji, Sheng Wu, and Chunxiao JiangAuthors Info & ClaimsPages 3414 - 3429
Abstract
With the development of commercial off-the-shelf hardware, low Earth orbit (LEO) satellites are promising to provide flexible edge computing services. In this paper, we investigate a digital twin (DT)-empowered satellite-terrestrial cooperative edge computing network, where computation tasks from terrestrial users can be partially offloaded to the associated base station (BS) edge server, the associated LEO satellite edge server, and an adjacent LEO satellite edge server. We formulate a multi-tier computation offloading optimization problem to minimize the weighted sum of total system delay and satellite energy consumption, where a LEO-layer problem and a DT-layer problem are involved. The LEO-layer problem optimizes the three-tier computation resource allocation and task splitting ratio. From the multi-satellite network perspective, the DT-layer problem optimizes how many resources will be shared between adjacent satellites. We then propose a multi-agent double actors twin delayed deterministic policy gradient (MA-DATD3) algorithm to optimize the LEO-layer problem, and adopt a centralized training and decentralized execution (CTDE) paradigm. The proposed MA-DATD3 algorithm is extended to solve the DT-layer problem in a centralized way, and the resource sharing between adjacent satellites is optimized to maximize the time-averaged reward. Simulation results show that our algorithm achieves a better performance than the MADDPG algorithm, and effectively improves the computation offloading performance while balancing the energy consumption and the total delay.
References
[1]
S. Cioni, R. De Gaudenzi, O. Del Rio Herrero, and N. Girault, “On the satellite role in the era of 5G massive machine type communications,” IEEE Netw., vol. 32, no. 5, pp. 54–61, Sep. 2018.
[2]
M. Shaat, E. Lagunas, A. I. Perez-Neira, and S. Chatzinotas, “Integrated terrestrial-satellite wireless backhauling: Resource management and benefits for 5G,” IEEE Veh. Technol. Mag., vol. 13, no. 3, pp. 39–47, Sep. 2018.
[3]
J. Liu, Y. Shi, Z. Md. Fadlullah, and N. Kato, “Space-air-ground integrated network: A survey,” IEEE Commun. Surveys Tuts., vol. 20, no. 4, pp. 2714–2741, 4th Quart., 2018.
[4]
M. Jia, X. Gu, Q. Guo, W. Xiang, and N. Zhang, “Broadband hybrid satellite-terrestrial communication systems based on cognitive radio toward 5G,” IEEE Wireless Commun., vol. 23, no. 6, pp. 96–106, Dec. 2016.
[5]
X. Zhu, C. Jiang, L. Kuang, N. Ge, and J. Lu, “Non-orthogonal multiple access based integrated terrestrial-satellite networks,” IEEE J. Sel. Areas Commun., vol. 35, no. 10, pp. 2253–2267, Oct. 2017.
[6]
J. Liu, Y. Shi, L. Zhao, Y. Cao, W. Sun, and N. Kato, “Joint placement of controllers and gateways in SDN-enabled 5G-satellite integrated network,” IEEE J. Sel. Areas Commun., vol. 36, no. 2, pp. 221–232, Feb. 2018.
[7]
Q. Tang, Z. Fei, B. Li, and Z. Han, “Computation offloading in LEO satellite networks with hybrid cloud and edge computing,” IEEE Internet Things J., vol. 8, no. 11, pp. 9164–9176, Jun. 2021.
[8]
K. B. Letaief, W. Chen, Y. Shi, J. Zhang, and Y. A. Zhang, “The roadmap to 6G: AI empowered wireless networks,” IEEE Commun. Mag., vol. 57, no. 8, pp. 84–90, Aug. 2019.
[9]
H.-C. Chao, D. E. Comer, and O. Kao, “Space and terrestrial integrated networks: Emerging research advances, prospects, and challenges,” IEEE Netw., vol. 33, no. 1, pp. 6–7, Jan. 2019.
[10]
R. Xie, Q. Tang, Q. Wang, X. Liu, F. R. Yu, and T. Huang, “Satellite-terrestrial integrated edge computing networks: Architecture, challenges, and open issues,” IEEE Netw., vol. 34, no. 3, pp. 224–231, May 2020.
[11]
Q.-V. Phamet al., “Aerial computing: A new computing paradigm, applications, and challenges,” IEEE Internet Things J., vol. 9, no. 11, pp. 8339–8363, Jun. 2022.
[12]
Q. Liet al., “Service coverage for satellite edge computing,” IEEE Internet Things J., vol. 9, no. 1, pp. 695–705, Jan. 2022.
[13]
Z. Li, C. Jiang, and L. Kuang, “Double auction mechanism for resource allocation in satellite MEC,” IEEE Trans. Cognit. Commun. Netw., vol. 7, no. 4, pp. 1112–1125, Dec. 2021.
[14]
N. Chenget al., “Space/aerial-assisted computing offloading for IoT applications: A learning-based approach,” IEEE J. Sel. Areas Commun., vol. 37, no. 5, pp. 1117–1129, May 2019.
[15]
J. Du, C. Jiang, H. Zhang, Y. Ren, and M. Guizani, “Auction design and analysis for SDN-based traffic offloading in hybrid satellite-terrestrial networks,” IEEE J. Sel. Areas Commun., vol. 36, no. 10, pp. 2202–2217, Oct. 2018.
[16]
S. Sthapit, S. Lakshminarayana, L. He, G. Epiphaniou, and C. Maple, “Reinforcement learning for security-aware computation offloading in satellite networks,” IEEE Internet Things J., vol. 9, no. 14, pp. 12351–12363, Jul. 2022.
[17]
X. Zhu and C. Jiang, “Delay optimization for cooperative multi-tier computing in integrated satellite-terrestrial networks,” IEEE J. Sel. Areas Commun., vol. 41, no. 2, pp. 366–380, Feb. 2023.
[18]
X. Caoet al., “Edge-assisted multi-layer offloading optimization of LEO satellite-terrestrial integrated networks,” IEEE J. Sel. Areas Commun., vol. 41, no. 2, pp. 381–398, Feb. 2023.
[19]
H. Zhang, L. Feng, X. Liu, K. Long, and G. K. Karagiannidis, “User scheduling and task offloading in multi-tier computing 6G vehicular network,” IEEE J. Sel. Areas Commun., vol. 41, no. 2, pp. 446–456, Feb. 2023.
[20]
S. Shen, Y. Ren, Y. Ju, X. Wang, W. Wang, and V. C. M. Leung, “EdgeMatrix: A resource-redefined scheduling framework for SLA-guaranteed multi-tier edge-cloud computing systems,” IEEE J. Sel. Areas Commun., vol. 41, no. 3, pp. 820–834, Mar. 2023.
[21]
A. M. Seid, G. O. Boateng, B. Mareri, G. Sun, and W. Jiang, “Multi-agent DRL for task offloading and resource allocation in multi-UAV enabled IoT edge network,” IEEE Trans. Netw. Service Manage., vol. 18, no. 4, pp. 4531–4547, Dec. 2021.
[22]
Z. Zhouet al., “Secure and latency-aware digital twin assisted resource scheduling for 5G edge computing-empowered distribution grids,” IEEE Trans. Ind. Informat., vol. 18, no. 7, pp. 4933–4943, Jul. 2022.
[23]
H. Liaoet al., “Cloud-edge-device collaborative reliable and communication-efficient digital twin for low-carbon electrical equipment management,” IEEE Trans. Ind. Informat., vol. 19, no. 2, pp. 1715–1724, Feb. 2023.
[24]
Y. Lu, X. Huang, K. Zhang, S. Maharjan, and Y. Zhang, “Communication-efficient federated learning for digital twin edge networks in industrial IoT,” IEEE Trans. Ind. Informat., vol. 17, no. 8, pp. 5709–5718, Aug. 2021.
[25]
R. Dong, C. She, W. Hardjawana, Y. Li, and B. Vucetic, “Deep learning for hybrid 5G services in mobile edge computing systems: Learn from a digital twin,” IEEE Trans. Wireless Commun., vol. 18, no. 10, pp. 4692–4707, Oct. 2019.
[26]
S. Boyd, S. P. Boyd, and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
[27]
J. Lyu, X. Ma, J. Yan, and X. Li, “Efficient continuous control with double actors and regularized critics,” 2021, arXiv:2106.03050.
[28]
S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” 2018, arXiv:1802.09477.
Index Terms
- Cooperative Multi-Agent Deep Reinforcement Learning for Computation Offloading in Digital Twin Satellite Edge Networks
Index terms have been assigned to the content through auto-classification.
Recommendations
Analysis of satellite-induced factors affecting the accuracy of the BDS satellite differential code bias
Differential code bias (DCB) is one of the main error sources of the positioning, navigation and timing services and slant total electron content extraction. The DCB can usually be estimated together with ionospheric model coefficients from the data of ...
Precise orbit determination for BDS-3 satellites using satellite-ground and inter-satellite link observations
Since November 2017, eight BeiDou global navigation system (BDS-3) satellites equipped with Ka-band inter-satellite link (ISL) payloads have been launched into medium earth orbit. We present the precise orbit determination (POD) for BDS-3 satellites ...
Deep Reinforcement Learning-based Task Offloading in Satellite-Terrestrial Edge Computing Networks
2021 IEEE Wireless Communications and Networking Conference (WCNC)In remote regions (e.g., mountain and desert), cellular networks are usually sparsely deployed or unavailable. With the appearance of new applications (e.g., industrial automation and environment monitoring) in remote regions, resource-constrained ...
Comments
Information & Contributors
Information
Published In
0733-8716 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 11 September 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in