Quantum Multi-Agent Reinforcement Learning for Cooperative Mobile Access in Space-Air-Ground Integrated Networks

Numerous projects focus on establishing wireless connections to create aerial NTN devices, including UAVs or satellites [21]. Given that these rely on battery-based energy management, minimizing energy consumption is crucial to stable operation in unknown environments for the efficient operation of multiple UAVs and satellites [22]. In the literature, the efficient operation of multiple UAVs has garnered significant attention [23]. Minimizing energy consumption is important to stable operation in unfamiliar environments, necessitating efficient communications [24]. At the same time, efficient scheduling among satellites is imperative to ensure swift responses to diverse sightings and unforeseen events [25]. UAVs, characterized by remarkable acquisition flexibility and very high spatial resolution (VHSR), and LEO satellites, capable of providing time-series data across extensive areas, have traditionally been employed independently. However, the proposed algorithm in [26] can minimize total energy costs and reduce time complexity which is crucial for optimizing their effective operation for both UAVs and satellites. Therefore UAVs and satellites must be controlled cooperatively to improve performance [27]. To efficiently manage both UAVs and satellites, numerous studies have demonstrated different methodologies for applying RL algorithms [28]. The proposed algorithm in [29] proves the superiority of RL, particularly beneficial in the management of multiple agents. However, to build global SAGIN mobile access, more agents need to be controlled [30]. Notably, quantum algorithms have advantages in managing large-scale scenarios, such as those encountered in aerial networks [31]. This paper demonstrates the superiority of using QRL over RL in multi-agent scheduling.

In QNN architectures, a significant deviation from classical neural networks is the utilization of qubits as the unit for basic learning computations [32]. Within quantum systems, qubits stand as the fundamental units of information, and their representation is grounded in the base states of $\left|0\right\rangle:=[1,0]^{T}$ and $\left|1\right\rangle:=[0,1]^{T}$. The representation of a single qubit state can be realized through a normalized 2D complex vector as $\left|\psi\right>=\alpha\left|0\right>+\beta\left|1\right>$ and $\left\|\alpha\right\|^{2}+\left\|\beta\right\|^{2}=1$ holds, where $\left\|\alpha\right\|^{2}$ and $\left\|\beta\right\|^{2}$ denote the probabilities of observing $\left|0\right>$ and $\left|1\right>$, respectively. The QNN computation is carried out over the 3D Bloch sphere, defined as the Hilbert space which represents the quantum domain. Expressing this within the Bloch sphere, which serves as a representation of the quantum domain, it can be geometrically denoted as, $\left|\psi\right>=\cos(\theta)\left|0\right>+e^{i\phi}\sin(\theta)\left|1\right>$, where $\theta$ denotes a parameter that determines the probabilities of measuring $\left|0\right>$ and $\left|1\right>$, and $\phi$ represents the relative phase, respectively, where $0\leq\theta\leq\pi$ and $0\leq\phi<2\pi$ [32]. Fig. 2(a) shows a qubit represented over the Bloch sphere. When considering a $q$ qubit system, the representation of quantum states within the system’s Hilbert space is as $\left|\psi\right\rangle=\sum^{2^{q}-1}_{l=0}\omega_{l}\left|l\right\rangle$, where $\left|\psi\right\rangle$ denotes the quantum state, $\left|l\right\rangle$ represents $l$-th basis, and $\omega_{l}$ stands for the probability amplitude of $q$ qubit system, respectively. Then, the probability amplitude fulfills $\sum^{2^{q}-1}_{l=0}|\omega_{l}|^{2}=1$. A significant component in classical neural networks is a hidden layer, capable of representing linear and nonlinear transformations to achieve accurate function approximation within the neural network. Hence, the primary design consideration factors in QNN involve designing and implementing linear and nonlinear transformations over the 3D sphere. This QNN design facilitates the fundamental enablement of QRL-based control, achieved by incorporating the states and actions of RL-based control as inputs and outputs within QNN architectures.

In QNN architecture, there are three primary components: (i) state encoding, (ii) parameterized quantum circuit (PQC), and (iii) measurement, as illustrated in Fig. 2(b).

• •

  State Encoding. The encoder performs the function of converting the classical data, represented as $\zeta_{t}$ at a specific time $t$, to the initialized quantum state $\left|0\right\rangle$. The encoder carries out this function due to the inability of quantum circuits to directly accept classical bits. Through the application of multiple unitary matrices, denoted as $U(\cdot)$, this encoding transformation is achieved mathematically. An important point to highlight is that the encoder does not include any trainable parameters. Thus, the encoded quantum state of the QNN at a specific time $t$ is defined as $\left|\psi_{0;t}\right\rangle=U_{\text{ENC}}(\zeta_{t})\left|0\right\rangle^{\otimes q}$, where the classical data $\zeta_{t}$ serves as rotation angles within the set of encoding gates $U$.

• •

  PQC. The operations performed by PQC are analogous to the multiplications seen in the accumulated hidden layers of classical neural networks. Quantum gates can transform the state of qubits through the operations they perform [32]. Within this paper, the following three gates will be introduced: Pauli, Controlled, and rotation gates [32]. Outlined below are the definitions for Pauli-$\Gamma$ gates and Controlled-$\Gamma$ gates, i.e., $X\!\!=\!\!\begin{bmatrix}0&1\\ 1&0\end{bmatrix}\!$, $\!Y\!\!=\!\!\begin{bmatrix}0&\!-\!i\\ i&0\end{bmatrix}\!$, $\!Z\!\!=\!\!\begin{bmatrix}1&0\\ 0&\!-\!1\end{bmatrix}\!$, and $C\Gamma\!\!=\!\!\begin{bmatrix}{\textbf{{I}}}&0\\ 0&\Gamma\end{bmatrix}$, where $i=\sqrt{(-1)}$, $\forall\Gamma\in\left\{X,Y,Z\right\}$, and I stands for the identity matrix, respectively. The Pauli-$\Gamma$ gates perform $180\,^{\circ}$ rotations of the quantum state in the x, y, and z axes of the Bloch sphere. Between two qubits, the Controlled-$\Gamma$ gates produce entanglement. Within QNN, rotation gates $R_{\Gamma}$ featuring the trainable parameters $\theta_{k}$, defined within the range $[0,2\pi]$, find widespread utilization. This can be represented as follows: $R_{\Gamma}(\theta_{k})=e^{-i\frac{\theta_{k}}{2}\Gamma}$. Achieving rotations and entanglement of all qubits involves utilizing Pauli-$\Gamma$, Controlled-$\Gamma$, and rotation gates. At this moment, Pauli-$\Gamma$ gates and $R_{\Gamma}$ are employed for implementing linear transformations, while the Controlled-$\Gamma$ gates are utilized for nonlinear transformations. Therefore, PQC achieves two transformations on the 3D sphere. Consequently, in PQC, it can vary depending on the configuration of the $R_{\Gamma}$ and Controlled-$\Gamma$ gates, and is an important factor in building a QNN. To thoroughly explore trainable rotation parameters and entanglement, we implement multiple quantum layers in this paper, each consisting of $R_{\Gamma}$ gates within PQC of each QNN. At a specific time $t$, the quantum state of the QNN, denoted as $\left|\psi_{t}\right\rangle$, can be represented as $\left|\psi_{t}\right\rangle=\prod_{l=1}^{L}\nolimits\boldsymbol{U}_{l}(\theta_{t})U_{\text{ENC}}(\zeta_{t})\left|0\right\rangle^{\otimes q}$, where $\boldsymbol{U}_{l}(\theta_{t})$ stands for the $l$-th quantum layer at the specific time $t$ with its corresponding set of trainable parameters. Observe that $\boldsymbol{U}_{l}(\theta_{t})$ takes the trainable parameters as inputs, therefore it works differently from the encoder’s gates.

• •

  Measurement. The quantum state that is acquired by PQC is utilized as the input for measurement. In this process, quantum data is decoded back to the original format before performing measurements on the input. The z-axis is commonly used for measurements, but axes in other directions can also be used if they are appropriately defined. The quantum state collapses and its properties become observable after the quantum state is measured. Upon completion of the decoding procedure, the observable property is employed to minimize the loss function. Achieving the expected decoded value of the quantum state $\left|\psi_{t}\right\rangle$ can be accomplished through $\left\langle\psi_{t}\right|O\left|\psi_{t}\right\rangle$, where $\left|\psi_{t}\right\rangle=\prod_{l=1}^{L}\boldsymbol{U}_{l}(\theta_{t})U_{\text{ENC}}(\zeta_{t})\left|0\right\rangle^{\otimes q}$, $\left\langle\psi_{t}\right|$ denotes the conjugate transpose of $\left|\psi_{t}\right\rangle$, and $O$ represents the observable, respectively.

This section investigates the use of QMARL for scheduling CubeSats and HALE-UAVs, presenting a strong argument for its preference over conventional MARL approaches. Conventional MARL has been effective for optimizing decisions in scenarios with relatively small action dimensions. Nonetheless, within intricate systems like integrated networks using CubeSats/HALE-UAVs, characterized by exponentially vast action dimensions, the efficacy of conventional MARL diminishes due to computational burden and the inefficacy in managing extensive action spaces. The expansion of the action dimension introduces the challenge of the curse of dimensionality [33], a significant impediment in conventional MARL frameworks. QMARL, empowered by quantum computing features such as superposition and entanglement, offers a significant computational edge [34]. This quantum advantage allows QMARL to efficiently process large-scale data and complex decision matrices [35], presenting a superior solution for the extensive action dimensions encountered in integrated networks using CubeSats/HALE-UAVs. Moreover, the multi-agent dynamics of these integrated networks involving many communicating devices such as multiple GSs, CubeSats, and HALE-UAVs make the scheduling decision-making problem more complex. QMARL signifies a crucial advancement in overcoming the challenges of high-dimensional and complex scheduling tasks for integrated networks using CubeSats/HALE-UAVs. Its enhanced computational strength and ability to effectively manage multi-agent scenarios establish it as a powerful and efficient approach, facilitating the development of more sophisticated, effective, and dependable SAGIN.

The considered global SAGIN is illustrated in Fig. 1 and structured around three principal elements, $N$ GSs, a fleet of $M$ CubeSats, and a group of $L$ HALE-UAVs. Each GS is denoted as $G_{i}$, $i\in\mathcal{N}$, and note that $|\mathcal{N}|\triangleq N$. In addition, CubeSats and HALE-UAVs are denoted as $S_{j}$ and $A_{l}$, respectively, where $S_{j},j\in\mathcal{M}$ and $A_{l},l\in\mathcal{L}$, and also note that $|\mathcal{M}|\triangleq M$ and $|\mathcal{L}|\triangleq L$. Our proposed scheduling works by each GS $G_{i}$ to establish the communications with CubeSats $S^{i}_{j}$ or HALE-UAVs $A^{i}_{l}$ that are located within the coverage of $G_{i}$, for network access services. The main purpose of this scheduling is for maximizing (i) the residual energy amounts of NTN devices, (ii) the fair energy consumption among NTN devices, and (iii) the global access performance in terms of capacity and QoS, in SAGIN systems.

In order to ensure the maneuvers of HALE-UAVs while maintaining the equilibrium among the energy levels of HALE-UAVs, energy expenditure modeling for HALE-UAV is essential. The required energy is the minimum energy amount to overcome aerodynamic drag and advance in each HALE-UAV. The energy is equivalent to the work per unit over time under the force applied to the dynamic system, and it is defined as the dot product of force and velocity. Therefore, the required energy of the $l$-th HALE-UAV at time $t$, denoted as $E^{A}_{l}(t)$, is defined as $E^{A}_{l}(t)=DV$, where $D$ and $V$ denote its drag and velocity at time $t$, respectively. Here, drag $D$ can be obtained as $D=\frac{1}{2}\rho V^{2}SC_{D}=qSC_{D}$, where $C_{D}$ is drag coefficient. Because $C_{D}$ is expressed as $C_{D}=C_{D_{0}}+kC_{L}^{2}$ and $C_{L}$ is expressed as $C_{L}=\frac{W}{\frac{1}{2}\rho V^{2}S}=\frac{W}{qS}$, the required energy of the $l$-th HALE-UAV at time $t$, i.e., $E^{A}_{l}(t)$, is,

where $C_{D_{0}}$, $\rho$, $V$, $S$, $k$, $W$, and $q$ are the parasite drag coefficient at zero lift, density of the air, velocity, wing surface area, induced drag coefficient, HALE-UAV weight, and dynamic pressure ($q=\frac{1}{2}\rho V^{2}$) [36], respectively. As expressed in (1), the required energy is composed of the parasite energy and induced energy [37]. Here, the parasite energy arises from parasite drag, encompassing skin friction drag (drag that varies with the UAV’s surface texture), form drag (drag that depends on the HALE-UAV’s size, structure, and shape), and interference drag (drag generated from the interaction between skin friction and form drag) [38]. In addition, the induced energy originates from the drag produced by generating lift. This type of drag is caused by wingtip vortices, resulting from the differential pressure on the wing’s upper and lower surfaces, which in turn creates downwash at the wing’s rear. Accordingly, $P_{p}$ increases with the cube of velocity, whereas $P_{i}$ is inversely related to velocity, demonstrating the dynamics of aerodynamic drag in relation to the UAV’s velocity [39].

On the other hand, velocity $V$ is computed as the aggregate of velocities along each axis, formulated as $V=\sqrt{u^{2}+v^{2}+w^{2}}$, where $u$, $v$, and $w$ represent the velocities over the $x$-, $y$-, and $z$-axes of body axis coordinate system, respectively. Here, velocity $V$ in (1) is the velocity based on the body axis coordinate system of aircraft. Nevertheless, due to the fact that the velocities of HALE-UAVs for each axis are determined with the relation to the ground coordinate system, it is imperative to utilize coordinate transformation matrices. Therefore, velocities $u_{1}$, $v_{1}$, and $w_{1}$ in the ground coordinate system are transformed into the velocities $u$, $v$, and $w$ within the body axis coordinate system through multiplication by the coordinate transformation matrices $L_{1}$, $L_{2}$, and $L_{3}$, which is expressed as,

where $L_{1}$, $L_{2}$, and $L_{3}$ are the transformation matrices over the $z$-axis, $y$-axis, and $x$-axes, sequentially. The geometric relationships among these transformations are illustrated in Fig. 3, and the transformation of coordinates for each axis can be articulated via,

where $\psi$, $\theta$, and $\phi$ represent the rotations over the $z$-, $y$-, and $z$-axes, respectively. Within the real flight environment of HALE-UAVs, such disturbances are attributable to turbulence and wind gusts, which have the potential to alter the UAV’s rotational orientation. Amidst conditions where turbulence and gusts are prevalent across all axes, the goal of HALE-UAV is to simultaneously optimize the global access performance of the integrated network and the energy use of HALE-UAV. Details pertaining to the HALE-UAV deployed in this paper are compiled in Table I.