H_∞ State-Feedback Control of Multi-Agent Systems with Data Packet Dropout in the Communication Channels: A Markovian Approach

Abstract

The paper presents an $H_{\infty }$ type control procedure for multi-agent systems taking into account possible data dropout in the communication network. The data dropout is modelled using a standard homogeneous Markov chain leading to an $H_{\infty }$ type control problem for stochastic multi-agent systems with Markovian jumps. The considered $H_{\infty }$ type criterion includes, besides the components corresponding to the attenuation condition of exogenous disturbance inputs, quadratic terms aiming to acquire the consensus between the agents. It is shown that in the case of identical agents, a state-feedback controller with Markov parameters may be determined solving two specific systems of Riccati equations whose dimension does not depend on the number of agents. Iterative procedures to solve such systems are also presented together with an illustrative numerical example.

1. Introduction

Multi-agent systems received a considerable interest in control engineering over the last decades due to their wide area of applications including terrestrial, maritime, aerial and space surveillance and monitoring missions. Some early developments and comprehensive surveys in this field may be found, for instance, in [1,2,3,4]. The design requirements for the control of multi-agent systems may be formulated from different perspectives, a lot of literature treating these topics being available these days. Although the review of the control design methodologies for multi-agent systems is beyond the purpose of this paper, one will mention however some monographs as [5,6,7,8], presenting different multi-agent control problems. An important aspect related the multi-agent systems is their distributed control which is characterised, in contrast with the centralised case, by the absence of a control decision-maker. Such formulations, primarily considered for agents with single and double integrators models (see, e.g., [9]) have been then investigated for more general linear systems, as in [10]. Consensus of nonlinear agents using output feedback has been analysed, for instance, in [11] and an algorithm based on feedback linearisation of nonlinear agents with output measurements may be found in [12]. In [13], a stability analysis from the perspective of a hybrid modelling is proposed using invariant sets and the Lyapunov stability theory. Consensus problems for Bernoulli networks have been considered for instance in [14] and for high-order multi-agent systems under deterministic and Markovian switching network topologies, in [15]. Other consensus problems in stochastic sense have been investigated, for instance, in [16] and in [17] under Markov switching networks for agents with first and second order dynamics. Many formulations of the control problems for multi-agent systems include optimisation criteria. As shown for instance, in [18], the complexity associated with the computation of distributed optimal controller significantly increases. In [19], the authors study the structural properties of optimal control problems with infinite-horizon linear-quadratic criteria by analysing the spatial structure of the solution to corresponding Lyapunov operator and Riccati equations. A problem of synthesising a distributed dynamic output feedback achieving $H_{\infty }$ performance is presented in [20]. In [21], a decentralised Markovian $H_{\infty }$ control problem is considered for first-order dynamics of the agents in presence of time-delay conditions; the optimal control is expressed in terms of the solutions of a system of linear matrix inequalities depending on the dynamics of the multi-agent system. An $H_{\infty }$ state-feedback consensus controller for undirected multi-agent systems is derived in [22] for a more general class of N identical agents; the complexity of the $H_{\infty }$ consensus problem is reduced by representing the problem by N number linear systems. In fact, optimal control problems of multi-agent systems remain a domain of interest due to the design and implementation complexity of the control laws, even in the case of identical agents (see, for instance, [22,23,24,25]). In [26], a linear-quadratic control problem for identical agents is considered, for which it is proved that the optimal solution depends on the number of agents and on the stabilising solutions of two Riccati equations having the same order as the agents dynamics. This conclusion shows that the computational requirements may be significantly reduced, especially for large-scale multi-agent systems. The research has been continued in [27] where the robustness properties of the decentralised linear-quadratic optimal controller are analysed.

More recently, an $H_{\infty }$ optimisation problem for identical stochastic linear models corrupted with multiplicative noises was formulated and solved in [28], aiming to provide disturbance attenuation performance together with robust stability with respect to parametric uncertainties in the agents models. Based on the state-feedback gains of the centralised state-feedback controller, a distributed controller depending on the adjacency matrix associated with the undirected graph of the communication network was obtained using the spectra of Lyapunov operators.

In this paper, the loss of links between agents is modelled by linear stochastic systems with Markovian jumps. The proposed methodology allows to consider different configurations of the network, each of them corresponding to a state of the Markov chain. Markov switching network models may be found in many papers between which one mentions [29,30,31]. The formulation and the developments presented in this paper considers stochastic models both for the agents and for the network. The $H_{\infty }$ cost function for the optimal control design includes, besides the expression for the attenuation of exogenous disturbance inputs, quadratic terms aiming to acquire the state consensus between the agents. The optimal $H_{\infty }$ state-feedback gains are expressed in terms of the stabilising solutions of two systems of coupled game-theoretic Riccati equations having the same order as the dynamics of a single agent. This coupling between the Riccati equations is typical in the optimal control of stochastic Markovian systems, depending on the elements of the stationary transition rate matrix.

The paper is organised as follows: the $H_{\infty }$ control problem for stochastic system with Markovian jumps is formulated and solved in Section 2. The optimal gains of the control law are expressed in terms of the stabilising solution of a system of coupled algebraic Riccati equations with indefinite sign. A convergent iterative algorithm to determine the stabilising solution of this system of coupled Riccati equations is also presented. The Section 3 analyses the case of multi-agent $H_{\infty }$ control. The main result of this section allows to determine the optimal control law for multi-agent systems solving two specific systems of coupled game-theoretic Riccati equations corresponding to the dimension of a single agent. In Section 4, the case of dropout data packages in the communication networks is discussed and illustrated by a numerical example for a large-scale multi-agent system with two states of the Markov chain. The paper ends with some concluding remarks.

2. ${\mathit{H}}_{\infty }$ Type Control for Stochastic Systems with Markovian Jumps; The Case of a Single Agent

Consider the linear stochastic system

$$\begin{array}{c}\hfill \begin{array}{ccc}\dot{x}\left(t\right)\hfill & =& A\left(\eta \left(t\right)\right)x\left(t\right)+B_1\left(\eta \left(t\right)\right)w\left(t\right)+B_2\left(\eta \left(t\right)\right)u\left(t\right)\hfill \\ y_1\left(t\right)\hfill & =& C\left(\eta \right(t\left)\right)x\left(t\right)+D\left(\eta \right(t\left)\right)u\left(t\right)\hfill \\ y_2\left(t\right)\hfill & =& x\left(t\right)\hfill \end{array}\end{array}$$

(1)

where $x\in {\mathcal{R}}^n$ denotes the state vector, $w\in {\mathcal{R}}^{m_1}$ is an exogenous input, $u\in {\mathcal{R}}^{m_2}$ stands for the control input, $y_1\in {\mathcal{R}}^{p_1}$ is the quality output and $y_2\in {\mathcal{R}}^n$ denotes the measured output. Throughout the paper $\eta \left(t\right),t\ge 0$ denotes a continuous Markov chain with the state space $\mathcal{D}=\{1,\dots ,d\}$ and with the probability transition matrix $P\left(t\right)=\left[p_{ij}\left(t\right)\right]=e^{\Pi t},i,j\in \mathcal{D},t\ge 0$ in which the stationary transition rate matrix of $\eta$ is $\Pi =\left[{\pi }_{ij}\right]$ with ${\sum }_{j=1}^d{\pi }_{ij}=0,i\in \mathcal{D}$ and ${\pi }_{ij}\ge 0$ if $i\ne j$.

The triple $\{\Omega ,\mathcal{F},\mathcal{P}\}$ denotes a given probability space, $E\left[x\right]$ stands for the expectation of the random variable x, $E\left[x|\mathcal{H}\right]$ represents the conditional expectation of x with respect to the $\sigma$-algebra $\mathcal{H}\subset \mathcal{F}$ and $E\left[x\right|\eta \left(t\right)=i]$ is the conditional expectation with respect to the event $\eta \left(t\right)=i$. In the following developments it will be assumed that $C^{\top }\left(i\right)D\left(i\right)=0$ and $D^{\top }\left(i\right)D\left(i\right)=I_{m_2},\forall i\in \mathcal{D}$. For invertible $D{\left(i\right)}^{\top }D\left(i\right)$, $i=1,\dots ,d$, if these assumptions are not accomplished one may perform the following change of the control variable u

$$\begin{array}{c}\hfill u\left(\eta \left(t\right)\right)=-{\left(D{\left(\eta \left(t\right)\right)}^{\top }D\left(\eta \left(t\right)\right)\right)}^{-1}D{\left(\eta \left(t\right)\right)}^{\top }C\left(\eta \left(t\right)\right)x\left(t\right)+{\left(D{\left(\eta \left(t\right)\right)}^{\top }D\left(\eta \left(t\right)\right)\right)}^{-\frac{1}{2}}\tilde{u}\left(t\right)\end{array}$$

for which one can easily check that with the new control variable $\tilde{u}$ the orthogonality condition $C{\left(i\right)}^{\top }D\left(i\right)=0$ holds and $D{\left(i\right)}^{\top }D\left(i\right)=I_{m_2}$, $i=1,\dots ,d$.

For a multi-agent system with N agents, the indexes $k,\ell =1,\dots ,N$ will be used to define the connection between the agents k and ℓ.

Some known definitions and results used in the following developments will be briefly reminded (more details and proofs may be found, for instance, in [32,33]).

Definition 1. 

The stochastic system with Markov parameters

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=A\left(\eta \left(t\right)\right)x\left(t\right)\end{array}$$

(2)

is called exponentially stable in mean square (ESMS) if there exists $\beta \ge 1$ and $\alpha >0$ such that $E\left[{|\Phi \left(t\right)|}^2|n\left(0\right)=i\right]\le \beta e^{-\alpha t},\forall t\ge 0,i\in \mathcal{D}$, where $\Phi \left(t\right)$ denotes the fundamental (random) solution of the differential system (2).

Proposition 1. 

The stochastic system (2) is ESMS if and only if there exist the matrices $X\left(i\right)>0$, $i=1,\dots ,d$ verifying the system of Lyapunov-type inequalities

$$\begin{array}{c}\hfill A^{\top }\left(i\right)X\left(i\right)+X\left(i\right)A\left(i\right)+\sum _{j=1}^d{\pi }_{ij}X\left(j\right)<0.\end{array}$$

(3)

Throughout the paper it is assumed that the system

$$\begin{array}{c}\hfill \begin{array}{ccc}\dot{x}\left(t\right)\hfill & =& A\left(\eta \left(t\right)\right)x\left(t\right)\hfill \\ y\left(t\right)\hfill & =& C\left(\eta \left(t\right)\right)x\left(t\right)\hfill \end{array}\end{array}$$

(4)

is stochastically detectable, namely there exist a set of matrices $H\left(i\right),i=1,...,d$ such that the system $\dot{x}\left(t\right)=\left(A\left(\eta \left(t\right)\right)+H\left(\eta \left(t\right)\right)C\left(\eta \left(t\right)\right)\right)x\left(t\right)$ is ESMS.

The proof of the next result may be found, for instance, in [33] (Theorem 7 of Chapter 3).

Proposition 2. 

If the system (4) is stochastically detectable and if the system of Lyapunov-type equations

$$\begin{array}{c}\hfill A^{\top }\left(i\right)X\left(i\right)+X\left(i\right)A\left(i\right)+C^{\top }\left(i\right)C\left(i\right)+\sum _{j=1}^d{\pi }_{ij}X\left(j\right)=0\end{array}$$

has a symmetric solution with $X\left(i\right)\ge 0,\forall i\in \mathcal{D}$, then it is ESMS.

Proposition 3. 

If $v:{\mathcal{R}}^n\times \mathcal{D}\to \mathcal{R}$ is a function of $C^1$ class for every $i\in \mathcal{D}$ then

$$\begin{array}{c}\hfill \begin{array}{c}E\left[v\left(x\right(t),\eta (t\left)\right)\left|\eta \right(0)=i\right]-v(x_0,i)=\\ E\left[{\int }_0^t\left\{x^{\top }\left(\tau \right)A^{\top }\left(\eta \left(\tau \right)\right)+\sum _{j=1}^{d}v(x\left(\tau \right),j){\pi }_{\eta \left(\tau \right)j}\right\}d\tau |\eta \left(0\right)=i\right],i\in \mathcal{D},t\ge 0,\end{array}\end{array}$$

where $x\left(t\right)$ is the solution of the system (2) with the initial condition $x_0$.

The main result of this section is the following theorem.

Theorem 1. 

If the system of coupled Riccati equations

$$\begin{array}{c}\hfill \begin{array}{c}{A}^{\top }\left(i\right)X\left(i\right)+X\left(i\right)A\left(i\right)+X\left(i\right)\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right)X\left(i\right)\\ +\sum _{j=1}^d{\pi }_{ij}X\left(j\right)+C^{\top }\left(i\right)C\left(i\right)=0\end{array}\end{array}$$

(5)

has a stabilizing solution $\left(X\left(1\right),...,X\left(d\right)\right)$ with $X\left(i\right)\ge 0,\forall i\in \mathcal{D}$ for a certain $\gamma >0$, namely if the stochastic system with Markov jumps

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=\left(A\left(\eta \left(t\right)\right)+\left({\gamma }^{-2}{B}_1\left(\eta \left(t\right)\right)B_1^{\top }\left(\eta \left(t\right)\right)-B_2\left(\eta \left(t\right)\right)B_2^{\top }\left(\eta \left(t\right)\right)\right)X\left(\eta \left(t\right)\right))\right)x\left(t\right)\end{array}$$

is ESMS, where

$$\begin{array}{c}\hfill F\left(\eta \left(t\right)\right):=-B_2^{\top }\left(\eta \left(t\right)\right)X\left(\eta \left(t\right)\right),\end{array}$$

(6)

then the state-feedback control law $u\left(t\right)=F\left(\eta \right(t\left)\right)x\left(t\right)$ stabilises the system (1) and

$$\begin{array}{c}\hfill E\left[{\int }_0^{\infty }\left(|y_1{\left(t\right)|}^2-{\gamma }^2{\left|w\left(t\right)\right|}^2\right)dt\right]\le 0\end{array}$$

(7)

for all $w\in L_{\eta }^2\left([0,\infty ),{\mathcal{R}}^{m_1}\right)$, where the quality output $y_1\left(t\right)$ is determined with the initial condition $x\left(0\right)=0$ of the system (1).

Proof. 

In order to prove that the state feedback gain (6) stabilises (1) one may firstly rewrite the Riccati system (5) as

$$\begin{array}{c}\hfill \begin{array}{c}{\left(A\left(i\right)+B_2\left(i\right)F\left(i\right)\right)}^{\top }X\left(i\right)+X\left(i\right)\left(A\left(i\right)+B_2\left(i\right)F\left(i\right)\right)+{\gamma }^{-2}X\left(i\right)B_1\left(i\right)B_1^{\top }\left(i\right)X\left(i\right)\\ X\left(i\right)B_2\left(i\right)B_2^{\top }\left(i\right)X\left(i\right)+\sum _{j=1}^d{\pi }_{ij}X\left(j\right)+C^{\top }\left(i\right)C\left(i\right)=0\end{array}\end{array}$$

with $X\left(i\right)\ge 0,i=1,\dots ,d$. On the other hand, since the system (4) is assumed stochastically detectable, it follows that the system

$$\begin{array}{c}\hfill \begin{array}{ccc}\dot{x}\left(t\right)\hfill & =& \left(A\left(\eta \left(t\right)\right)+B_2\left(\eta \left(t\right)\right)F\left(\eta \left(t\right)\right)\right)x\left(t\right)\hfill \\ y\left(t\right)\hfill & =& \left[\begin{array}{c}{\gamma }^{-1}{B}_1^{\top }\left(\eta \left(t\right)\right)X\left(\eta \left(t\right)\right)\\ B_2^{\top }\left(\eta \left(t\right)\right)X\left(\eta \left(t\right)\right)\\ C\left(\eta \right(t\left)\right)\end{array}\right]x\left(t\right)\hfill \end{array}\end{array}$$

is also stochastically detectable and then, based on Proposition 2, one concludes that the above stochastic system with the state matrix $A\left(\eta \left(t\right)\right)+B_2\left(\eta \left(t\right)\right)F\left(\eta \left(t\right)\right)$ is ESMS. Since the system (1) with the control law $u\left(t\right)=F\left(\eta \right(t\left)\right)x\left(t\right)$ has the same state matrix it follows that the state-feedback (6) is stabilising.

In order to prove the last part of the theorem one introduces the function $V(x\left(t\right),\eta \left(t\right))=x^{\top }\left(t\right)X\left(\eta \left(t\right)\right)x\left(t\right)$. Using Proposition 3 for a certain initial condition $x_0$, it follows that

$$\begin{array}{c}\hfill \begin{array}{c}E\left[\left(x^{\top }\left(t\right)X\left(\eta \left(t\right)\right)x\left(t\right)-x_0^{\top }X\left(\eta \left(0\right)\right)x_0\right)|\eta \left(0\right)=i\right]\\ =E\left[{\int }_0^{\top }\left\{2{\left(A\left(\eta \left(\tau \right)\right)x\left(\tau \right)+B_1\left(\eta \left(\tau \right)\right)w\left(\tau \right)+B_2\left(\eta \left(\tau \right)\right)u\left(\tau \right)\right)}^{\top }X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right.\right.\\ \left.\left.+\sum _{j=1}^d{\pi }_{\eta \left(\tau \right)j}{x}^{\top }\left(\tau \right)X\left(j\right)x\left(\tau \right)\right\}d\tau |\eta \left(0\right)=i\right].\end{array}\end{array}$$

Adding $J(i,w,u):=E\left[{\int }_0^t\left(|y_1{\left(\tau \right)|}^2-{\gamma }^2{\left|w\left(\tau \right)\right|}^2\right)d\tau |\eta \left(0\right)=i\right],i\in \mathcal{D}$ and using (5) one obtains

$$\begin{array}{c}\hfill \begin{array}{c}J(i,w,u)+E\left[\left(x^{\top }\left(t\right)X\left(\eta \left(t\right)\right)x\left(t\right)-x_0^{\top }X\left(\eta \left(0\right)\right)x_0\right)|\eta \left(0\right)=i\right]\hfill \\ =E\left[{\int }_0^t\left\{x^{\top }\left(\tau \right)C^{\top }\left(\eta \left(\tau \right)\right)C\left(\eta \left(\tau \right)\right)x\left(\tau \right)+u^{\top }\left(\tau \right)u\left(\tau \right)-{\gamma }^2{w}^{\top }\left(\tau \right)w\left(\tau \right)\right.\right.\hfill \\ +x^{\top }\left(\tau \right)\left(A^{\top }(\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)+X\left(\eta \left(\tau \right)\right)A\left(\eta \left(\tau \right)\right)\right)x\left(\tau \right)+w^{\top }\left(\tau \right)B_1^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\hfill \\ +x^{\top }\left(\tau \right)X\left(\eta \left(\tau \right)\right)B_1\left(\eta \left(\tau \right)\right)w\left(\tau \right)+u^{\top }\left(\tau \right)B_2^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\hfill \\ \left.\left.+x^{\top }\left(\tau \right)X\left(\eta \left(\tau \right)\right)B_2\left(\eta \left(\tau \right)\right)u\left(\tau \right)+\sum _{j=1}^d{\pi }_{\eta \left(\tau \right)j}{x}^{\top }\left(\tau \right)X\left(j\right)x\left(\tau \right)\right\}d\tau |\eta \left(0\right)=i\right]\hfill \\ =E\left[{\int }_0^t\left\{x^{\top }\left(\tau \right)\left(A^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)+X\left(\eta \left(\tau \right)\right)A\left(\eta \left(\tau \right)\right)\right.\right.\right.\hfill \\ +X\left(\eta \left(\tau \right)\right)\left({\gamma }^{-2}{B}_1\left(\eta \left(\tau \right)\right)B_1^{\top }\left(\eta \left(\tau \right)\right)-B_2\left(\eta \left(\tau \right)\right)B_2^{\tau }\left(\eta \left(\tau \right)\right)\right)X\left(\eta \left(\tau \right)\right)\hfill \\ \left.+\sum _{j=1}^d{\pi }_{\eta \left(\tau \right)j}{x}^{\top }\left(\tau \right)X\left(j\right)+C^{\top }\left(\eta \left(\tau \right)\right)C\left(\eta \left(\tau \right)\right)\right)x\left(\tau \right)\hfill \\ +{\left(u\left(\tau \right)+B_2^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)}^{\top }\left(u\left(\tau \right)+B_2^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)\hfill \\ \left.\left.-\left(\gamma w\left(\tau \right)-{\gamma }^{-1}{B}_1^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right){\left(\gamma w\left(\tau \right)-{\gamma }^{-1}{B}_1^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)}^{\top }\right\}d\tau |\eta \left(0\right)=i\right].\hfill \end{array}\end{array}$$

Taking into account (5), one obtains

$$\begin{array}{c}\hfill \begin{array}{c}J(i,w,u)+E\left[\left(x^{\top }\left(t\right)X\left(\eta \left(t\right)\right)x\left(t\right)-x_0^{\top }X\left(\eta \left(0\right)\right)x_0\right)|\eta \left(0\right)=i\right]\hfill \\ =E\left[{\int }_0^t\left\{{\left(u\left(\tau \right)+B_2^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)}^{\top }\left(u\left(\tau \right)+B_2^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)\right.\right.\hfill \\ \left.\left.-\left(\gamma w\left(\tau \right)-{\gamma }^{-1}{B}_1^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right){\left(\gamma w\left(\tau \right)-{\gamma }^{-1}{B}_1^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)}^{\top }\right\}d\tau |\eta \left(0\right)=i\right].\hfill \end{array}\end{array}$$

For $t\to \infty$ and $u\left(t\right)=F\left(\eta \right(t\left)\right)x\left(t\right)$, from the above equation it follows that

$$\begin{array}{c}\hfill \begin{array}{c}E\left[{\int }_0^{\infty }\left(|y_1{\left(t\right)|}^2-{\gamma }^2{\left|w\left(t\right)\right|}^2\right)dt\right]=\sum _{i=1}^d{p}_i\left(0\right)x_0^{\top }X\left(i\right)x_0\\ -\sum _{i=1}^d{p}_i\left(0\right)E\left[{\int }_0^{\infty }\left(\gamma w\left(\tau \right)-{\gamma }^{-1}{B}_1^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)\right.\\ \left.\times {\left(\gamma w\left(\tau \right)-{\gamma }^{-1}{B}_1^{\top }\left(\eta \left(\tau \right)\right)X\left(\eta \left(\tau \right)\right)x\left(\tau \right)\right)}^{\top }d\tau |\eta \left(0\right)=i\right]\end{array}\end{array}$$

where $p_i\left(0\right):=\mathcal{P}\left(\eta \left(0\right)=i\right)$. Then, for $x_0=0$, the inequality (7) directly follows and it becomes equality for $w^{*}\left(t\right)={\gamma }^{-2}{B}_1^{T}X\left(\eta \left(t\right)\right)x\left(t\right)$. □

Remark 1. 

Matrix Riccati equations with indefinite sign as in the system (5) appear in $H_{\infty }$ control ([34]) and in mixed $H_2/H_{\infty }$ control problems ([35]) in the deterministic framework.

The next result proved in [36] gives a numerical procedure to compute the stabilising solution $\left(X\left(1\right),\dots ,X\left(d\right)\right)$ of the system of game theoretic Riccati-type Equation (5) assuming that such a solution exists.

Proposition 4. 

Assume that the system (1) is stochastically detectable and that the system of Riccati Equation (5) has a stabilising solution. Then the sequences ${\left\{X_k\left(i\right)\right\}}_{k\ge 0},{\left\{Z_k\left(i\right)\right\}}_{k\ge 0}$ defined by $X_0\left(i\right)=0$ and $X_{k+1}\left(i\right)=X_k\left(i\right)+Z_k\left(i\right)$, $i=1,\dots ,d$, where $Z_0\left(i\right)$ are the stabilising solutions of the system of Riccati type equations

$$\begin{array}{c}\hfill \begin{array}{c}{\left(A\left(i\right)+\frac{1}{2}{\pi }_{ii}{I}_n\right)}^{\top }{Z}_0\left(i\right)+Z_0\left(i\right)\left(A\left(i\right)+\frac{1}{2}{\pi }_{ii}{I}_n\right)\\ -Z_0\left(i\right)B_2\left(i\right)B_2^{\top }\left(i\right)Z_0\left(i\right)+C^{\top }\left(i\right)C\left(i\right)+\sum _{j=1,j\ne i}^d{Z}_0\left(i\right)=0,\end{array}\end{array}$$

(8)

$Z_k\left(i\right),k\ge 1$ are the stabilising solutions of the un-coupled Riccati equations

$$\begin{array}{c}\hfill M_k^{\top }\left(i\right)Z_k\left(i\right)+Z_k\left(i\right)M_k\left(i\right)-Z_k\left(i\right)B_2\left(i\right)B_2^{\top }\left(i\right)Z_k\left(i\right)+R_k\left(i\right)=0,\end{array}$$

and where

$$\begin{array}{c}\hfill \begin{array}{ccc}{M}_k\left(i\right)\hfill & =& A\left(i\right)+\frac{1}{2}{\pi }_{ii}{I}_n+\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right)X_k\left(i\right)\hfill \\ R_k\left(i\right)\hfill & =& {\gamma }^{-2}{Z}_{k-1}\left(i\right)B_1\left(i\right)B_1^{\top }\left(i\right)Z_{k-1}\left(i\right)+\sum _{j=1,j\ne i}^d{\pi }_{ij}{Z}_{k-1}\left(j\right),\hfill \end{array}\end{array}$$

are convergent and the limit of $X_k\left(i\right),i=1,\dots ,d$ when $k\to \infty$ is the stabilising solution of the system (5).

An iterative algorithm to solve the system of coupled Riccati equations with definite sign is given for completeness in Appendix A. The proof of the algorithm convergence may be found in [33] (Theorem 21 of Chapter 4).

3. Markovian ${\mathit{H}}_{\infty }$ Controller Design for Multi-Agent Systems

Consider $N>1$ agents with identical dynamics of form

$$\begin{array}{c}\hfill \begin{array}{ccc}{\dot{x}}_k\left(t\right)\hfill & =& A\left(\eta \left(t\right)\right)x_k\left(t\right)+B_1\left(\eta \left(t\right)\right)w_k\left(t\right)+B_2\left(\eta \left(t\right)\right)u_k\left(t\right)\hfill \\ y_{1k}\left(t\right)\hfill & =& C\left(\eta \left(t\right)\right)x_k\left(t\right)+D\left(\eta \left(t\right)\right)u_k\left(t\right)\hfill \\ y_{2k}\left(t\right)\hfill & =& x_k\left(t\right),t\ge 0,k=1,\dots ,N\hfill \end{array}\end{array}$$

(9)

with $C^{\top }\left(i\right)D\left(i\right)=0$ and $D^{\top }\left(i\right)D\left(i\right)=I_{m_1}$, $i=1,\dots ,d,$ and $k=1,\dots ,N$.

Remark 2. 

Although in (9) one considered the same standard homogeneous Markov chain for all agents one may also treat the case when each agent is modelled with its own stochastic process ${\eta }_k\left(t\right)$. Indeed, if each agent dynamics is modelled with a standard homogenous Markov chain ${\eta }_k\left(t\right),k=1,...,N$ with d states then one may consider for the multi-agent system (9) an extended Markov chain with $d^N$ states. Since in the present paper the Markov parameters are used to characterise the availability or the link failure between agents it follows that the maximal number of states of the Markov chain in these applications is $2^N$.

The dynamics of the multi-agent system (9) may be written in the following compact form

$$\begin{array}{c}\hfill \begin{array}{ccc}\dot{\tilde{x}}\left(t\right)\hfill & =& \tilde{A}\left(\eta \left(t\right)\right)\tilde{x}\left(t\right)+{\tilde{B}}_1\left(\eta \left(t\right)\right)\tilde{w}\left(t\right)+{\tilde{B}}_2\left(\eta \left(t\right)\right)\tilde{u}\left(t\right)\hfill \\ {\tilde{y}}_1\left(t\right)\hfill & =& \tilde{C}\left(\eta \left(t\right)\right)\tilde{x}\left(t\right)+\tilde{D}\left(\eta \left(t\right)\right)\tilde{u}\left(t\right)\hfill \\ {\tilde{y}}_2\left(t\right)\hfill & =& \tilde{x}\left(t\right)\hfill \end{array}\end{array}$$

(10)

in which $\tilde{x}:={\left[x_1^{\top },\dots ,x_N^{\top }\right]}^{\top }$, $\tilde{w}:={\left[w_1^{\top },\dots ,w_N^{\top }\right]}^{\top }$, $\tilde{u}:=[u_1^{\top },\dots ,$$u_N^{\top }{]}^{\top }$, ${\tilde{y}}_1:={\left[y_{11}^{\top },\dots ,y_{1N}^{\top }\right]}^{\top }$ and $\tilde{A}\left(\eta \left(t\right)\right):=I_N\otimes A\left(\eta \left(t\right)\right)$, ${\tilde{B}}_1\left(\eta \left(t\right)\right):=I_N\otimes B_1\left(\eta \left(t\right)\right)$, ${\tilde{B}}_2\left(\eta \left(t\right)\right):=I_N\otimes B_2\left(\eta \left(t\right)\right)$, $\tilde{C}\left(\eta \left(t\right)\right):=I_N\otimes C\left(\eta \left(t\right)\right)$, $\tilde{D}\left(\eta \left(t\right)\right):=I_N\otimes D\left(\eta \left(t\right)\right)$ where ⊗ denotes the Kronecker product.

For $\gamma >0$ define the cost function

$$\begin{array}{c}\hfill \begin{array}{c}J\left(\tilde{w},\tilde{u}\right)=E\left[{\int }_0^{\infty }\left\{|{\tilde{y}}_1{\left(t\right)|}^2-{\gamma }^2{\left|\tilde{w}\left(t\right)\right|}^2\right.\right.\hfill \\ \left.\left.+\frac{1}{2}\sum _{k=1}^N\sum _{\ell =1,\ell \ne k}^N{\left(x_k\left(t\right)-x_{\ell }\left(t\right)\right)}^{\top }{Q}_{k\ell }\left(\eta \left(t\right)\right)\left(x_k\left(t\right)-x_{\ell }\left(t\right)\right)\right\}dt\right]\hfill \end{array}\end{array}$$

(11)

where $Q_{k\ell }\left(i\right),k,\ell =1,\dots ,N$ and $i=1,\dots ,d$ are positive semidefinite weighting matrices. Then (11) may be rewritten as

$$\begin{array}{c}\hfill J\left(\tilde{w},\tilde{u}\right)=E\left[{\int }_0^{\infty }\left({\tilde{y}}_1{\left(t\right)}^{\top }{\tilde{y}}_1\left(t\right)-{\gamma }^2\tilde{w}{\left(t\right)}^{\top }\tilde{w}\left(t\right)+\tilde{x}{\left(t\right)}^{\top }\tilde{Q}\left(\eta \left(t\right)\right)\tilde{x}\left(t\right)\right)dt\right]\end{array}$$

(12)

in which $\tilde{Q}\left(i\right)$ has the block elements

$$\begin{array}{c}\hfill \begin{array}{ccc}{\tilde{Q}}_{kk}\left(i\right)\hfill & =& C^{\top }\left(i\right)C\left(i\right)+\sum _{\ell =1,\ell \ne k}^N{Q}_{k\ell }\left(i\right),\hfill \\ {\tilde{Q}}_{k\ell }\left(i\right)\hfill & =& -Q_{k\ell }\left(i\right),k,\ell =1,\dots ,N,k\ne \ell ,\hfill \end{array}\end{array}$$

(13)

$i=1,\dots ,d.$ Choosing $Q_{k\ell }\left(i\right)=P^{\top }\left(i\right)P\left(i\right),k,\ell =1,\dots ,N,k\ne \ell$, $i=1,\dots ,d$, it follows that the block elements of the matrix $\tilde{Q}\left(i\right)$ are

$$\begin{array}{c}\hfill \begin{array}{ccc}{\tilde{Q}}_{kk}\left(i\right)\hfill & =& C^{\top }\left(i\right)C\left(i\right)+(N-1)P^{\top }\left(i\right)P\left(i\right),\hfill \\ {\tilde{Q}}_{k\ell }\left(i\right)\hfill & =& -P^{\top }\left(i\right)P\left(i\right),k,\ell =1,\dots ,N,k\ne \ell ,\hfill \end{array}\end{array}$$

$i=1,\dots ,d.$ One can easily check that

$$\begin{array}{c}\hfill {\tilde{y}}_1{\left(t\right)}^{\top }{\tilde{y}}_1\left(t\right)+\tilde{x}\left(t\right)\tilde{Q}\left(\eta \left(t\right)\right)\tilde{x}\left(t\right)=\tilde{\mathcal{C}}\left(\eta \left(t\right)\right)\tilde{x}\left(t\right)+\tilde{\mathcal{D}}\left(\eta \left(t\right)\right)\tilde{u}\left(t\right)\end{array}$$

where

$$\tilde{\mathcal{C}}\left(i\right):=\left[\begin{array}{cccc}& & \tilde{\mathcal{P}}\left(i\right)& \\ C\left(i\right)& 0& \dots & 0\\ 0& C\left(i\right)& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & C\left(i\right)\end{array}\right],\tilde{\mathcal{D}}\left(i\right):=\left[\begin{array}{cccc}& & 0_{n·N\times m_2·N}& \\ D\left(i\right)& 0& \dots & 0\\ 0& D\left(i\right)& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & D\left(i\right)\end{array}\right],$$

(14)

$i=1,...,d$ with $\tilde{\mathcal{P}}\left(i\right)$ satisfying the condition

$$\begin{array}{c}\hfill \tilde{\mathcal{P}}{\left(i\right)}^{\top }\tilde{\mathcal{P}}\left(i\right)=\left[\begin{array}{cccc}(N-1)P^{\top }\left(i\right)P\left(i\right)& -P^{\top }\left(i\right)P\left(i\right)& \dots & -P^{\top }\left(i\right)P\left(i\right)\\ -P^{\top }\left(i\right)P\left(i\right)& (N-1)P^{\top }\left(i\right)P\left(i\right)& \dots & -P^{\top }\left(i\right)P\left(i\right)\\ ⋮& ⋮& \ddots & ⋮\\ -P^{\top }\left(i\right)P\left(i\right)& -P^{\top }\left(i\right)P\left(i\right)& \dots & (N-1)P^{\top }\left(i\right)P\left(i\right)\end{array}\right],\end{array}$$

$i=1,\dots ,d$. Therefore the cost function (11) may be rewritten as

$$\begin{array}{c}\hfill J(\tilde{w},\tilde{u})=E\left[{\int }_0^{\infty }\left({\tilde{z}}^{\top }\left(t\right)\tilde{z}\left(t\right)-{\gamma }^2{\tilde{w}}^{\top }\left(t\right)\tilde{w}\left(t\right)\right)dt\right]\end{array}$$

(15)

where $z\left(t\right)=\tilde{\mathcal{C}}\left(\eta \left(t\right)\right)\tilde{x}\left(t\right)+\tilde{\mathcal{D}}\left(\eta \left(t\right)\right)\tilde{u}\left(t\right)$. Moreover, since it was assumed that $C^{\top }\left(i\right)D\left(i\right)=0$ and $D^{\top }\left(i\right)D\left(i\right)=I_{m_2},i=1,\dots ,d$ it follows that ${\tilde{\mathcal{C}}}^{\top }\left(i\right)\tilde{\mathcal{D}}\left(i\right)=0$ and ${\tilde{\mathcal{D}}}^{\top }\left(i\right)\tilde{\mathcal{D}}\left(i\right)=I_{m_1·N},i=1,\dots ,d$ and therefore one may apply Theorem 1 for the Markov stochastic system

$$\begin{array}{c}\hfill \begin{array}{ccc}\dot{\tilde{x}}\left(t\right)\hfill & =& \tilde{A}\left(\eta \left(t\right)\right)x\left(t\right)+{\tilde{B}}_1\left(\eta \left(t\right)\right)w\left(t\right)+{\tilde{B}}_2\left(\eta \left(t\right)\right)u\left(t\right)\hfill \\ \tilde{z}\left(t\right)\hfill & =& \tilde{\mathcal{C}}\left(\eta \left(t\right)\right)\tilde{x}\left(t\right)+\tilde{\mathcal{D}}\left(\eta \left(t\right)\right)\tilde{u}\left(t\right)\hfill \\ {\tilde{y}}_2\left(t\right)\hfill & =& \tilde{x}\left(t\right)\hfill \end{array}\end{array}$$

(16)

with the cost function (15).

The main result of this section is given by the following theorem.

Theorem 2. 

(i) If the system of coupled Riccati equations

$$\begin{array}{c}\hfill \begin{array}{c}{\tilde{A}}^{\top }\left(i\right)\tilde{X}\left(i\right)+\tilde{X}\left(i\right)\tilde{A}\left(i\right)+\tilde{X}\left(i\right)\left({\gamma }^{-2}{\tilde{B}}_1\left(i\right){\tilde{B}}_1^{\top }\left(i\right)-{\tilde{B}}_2\left(i\right){\tilde{B}}_2^{\top }\left(i\right)\right)\tilde{X}\left(i\right)\\ +\sum _{j=1}^d{\pi }_{ij}\tilde{X}\left(j\right)+{\tilde{Q}}^{\top }\left(i\right)\tilde{Q}\left(i\right)=0,i=1,\dots ,d.\end{array}\end{array}$$

(17)

has a stabilising solution $\left(\tilde{X}\left(1\right),\dots ,\tilde{X}\left(d\right)\right)$ with $\tilde{X}\left(i\right)\ge 0,i=1,\dots ,d$ then the stochastic system with Markov parameters

$$\begin{array}{c}\hfill \dot{\tilde{x}}\left(t\right)=\left(\tilde{A}\left(\eta \left(t\right)\right)+{\tilde{B}}_2\left(\eta \left(t\right)\right)\tilde{F}\left(\eta \left(t\right)\right)\right)x\left(t\right)+{\tilde{B}}_1\left(\eta \left(t\right)\right)\tilde{w}\left(t\right)\end{array}$$

(18)

where $\tilde{F}\left(i\right)=-{\tilde{B}}_2^{\top }\left(i\right)\tilde{X}\left(i\right)$, $i=1,\dots ,d$, is ESMS and for the initial condition $\tilde{x}\left(0\right)=0$,

$$\begin{array}{c}\hfill E\left[{\int }_0^{\infty }\left(|{\tilde{y}}_1{\left(t\right)|}^2-{\gamma }^2{\left|\tilde{w}\left(t\right)\right|}^2\right)dt\right]\le 0\end{array}$$

for all $\tilde{w}\in L_{\eta }^2\left([0,\infty ),{\mathcal{R}}^{N·m_1}\right)$.

(ii) The solution of (17) has the following structure

$$\begin{array}{c}\hfill \begin{array}{ccc}\tilde{X}\left(i\right)\hfill & :=& {\left[{\tilde{X}}_{k\ell }\right]}_{k,\ell =1,\dots ,N}where\hfill \\ {\tilde{X}}_{kk}\left(i\right)\hfill & :=& X_1\left(i\right)+(N-1)X_2\left(i\right)\hfill \\ {\tilde{X}}_{k\ell }\left(i\right)\hfill & :=& -X_2\left(i\right),k,\ell =1,\dots ,N,k\ne \ell ,\hfill \end{array}\end{array}$$

(19)

in which $\left(X_1\left(1\right),\dots ,X_1\left(d\right)\right)$ and $\left(X_2\left(1\right),\dots ,X_2\left(d\right)\right)$ are the solutions of the Riccati type equations

$$\begin{array}{c}\hfill \begin{array}{c}{A}^{\top }\left(i\right)X_1\left(i\right)+X_1\left(i\right)A\left(i\right)+X_1\left(i\right)\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right)X_1\left(i\right)\\ +\sum _{j=1}^d{\pi }_{ij}{X}_1\left(j\right)+C^{\top }\left(i\right)C\left(i\right)=0,i=1,\dots ,d\end{array}\end{array}$$

(20)

and

$$\begin{array}{c}\hfill \begin{array}{c}{\left[A\left(i\right)+\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right)X_1\left(i\right)\right]}^{\top }{X}_2\left(i\right)\\ +X_2\left(i\right)\left[(A\left(i\right)+\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right)X_1\left(i\right)\right]\\ +N{X}_2\left(i\right)\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right)X_2\left(i\right)\\ +\sum _{j=1}^d{\pi }_{ij}{X}_2\left(j\right)+P^{\top }\left(i\right)P\left(i\right)=0,i=1,\dots ,d,\end{array}\end{array}$$

(21)

respectively.

(iii) If the Riccati type systems (20) and (21) have the stabilising solutions $\left(X_1\left(1\right),\dots ,X_1\left(d\right)\right)$ and $\left(X_2\left(1\right),\dots ,X_2\left(d\right)\right)$ respectively, with $X_1\left(i\right)\ge 0$ and $X_2\left(i\right)\ge 0$, $i=1,\dots ,d$ then $\left(\tilde{X}\left(1\right),...,\tilde{X}\left(d\right)\right)$ with $\tilde{X}\left(i\right)$ defined in (19), is the stabilising solution of (17) and $\tilde{X}\left(i\right)\ge 0,i=1,\dots ,d$.

Proof. 

Part (i) of the statement is a direct consequence of Theorem 1.

The proof of (ii) is inspired by the arguments given in Theorems 1 and 2 of [26] for the multi agent linear quadratic control problem in deterministic framework. Thus, the stabilising solution $\left(\tilde{X}\left(1\right),\dots ,\tilde{X}\left(d\right)\right)$ of (17) has an $N\times N$ blocks structure, each of them having the size $(n\times n)$. Since all the matrix coefficients in (17) are diagonal it follows that the diagonal and the off-diagonal elements of $\tilde{X}\left(i\right),i=1,\dots ,d$ are equal, respectively. Then the diagonal block elements will be denoted by ${\tilde{X}}_1\left(i\right)$ and the off-diagonal ones by ${\tilde{X}}_2\left(i\right),i=1,\dots ,d$. The blocks $(k,\ell ),k,\ell \in \{1,\dots ,N\}$ of (17) have the form

$$\begin{array}{c}\hfill \begin{array}{c}{A}^{\top }\left(i\right){\tilde{X}}_{k\ell }\left(i\right)+{\tilde{X}}_{k\ell }\left(i\right)A\left(i\right)+\sum _{m=1}^N{\tilde{X}}_{km}\left(i\right)\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right){\tilde{X}}_{m\ell }\left(i\right)\\ +{\tilde{Q}}_{kl}\left(i\right)+\sum _{j=1}^d{\pi }_{ij}{\tilde{X}}_{k\ell }\left(j\right)=0.\end{array}\end{array}$$

Denoting $X_1\left(i\right):={\tilde{X}}_1\left(i\right)+(N-1){\tilde{X}}_2\left(i\right),i=1,\dots ,d$ and summing up the terms of the k-th row of (17) it follows that $X_1\left(i\right),i=1,\dots ,d$ verifies (20). Further, for any off-diagonal block $(k,\ell )$ with $k\ne \ell$ and $k,\ell \in \{1,\dots ,N\}$, direct computations give

$$\begin{array}{c}\hfill \begin{array}{c}{\left(A\left(i\right)+\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right){\tilde{X}}_1\left(i\right)\right)}^{\top }{\tilde{X}}_2\left(i\right)\\ +{\tilde{X}}_2\left(i\right)\left(A\left(i\right)+\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right){\tilde{X}}_1\left(i\right)\right)\\ +(N-2){\tilde{X}}_2\left(i\right)\left({\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)\right){\tilde{X}}_2\left(i\right)\\ +{\tilde{Q}}_{k\ell }\left(i\right)+\sum _{j=1}^d{\pi }_{ij}{\tilde{X}}_2\left(j\right)=0.\end{array}\end{array}$$

Using the fact that ${\tilde{X}}_1\left(i\right)=X_1\left(i\right)-(N-1){\tilde{X}}_2\left(i\right)$ and that ${\tilde{Q}}_{k\ell }\left(i\right)=-P^{\top }\left(i\right)P\left(i\right),i=1,\dots ,d$ from the above equation it follows that $X_2\left(i\right):=-{\tilde{X}}_2\left(i\right)$ is a solution of (21).

For part (iii) of the statement, one will prove that the stochastic system

$$\begin{array}{c}\hfill \dot{\tilde{x}}\left(t\right)=\left(\tilde{A}\left(\eta \left(t\right)\right)+\left({\gamma }^{-2}{\tilde{B}}_1\left(\eta \left(t\right)\right){\tilde{B}}_1^{\top }\left(\eta \left(t\right)\right)-{\tilde{B}}_2\left(\eta \left(t\right)\right){\tilde{B}}_2^{\top }\left(\eta \left(t\right)\right)\right)\tilde{X}\left(\eta \left(t\right)\right)\right)\tilde{x}\left(t\right)\end{array}$$

(22)

is ESMS, namely there exist the positive definite matrices $\tilde{S}\left(i\right),i=1,\dots ,d$ such that

$$\begin{array}{c}\hfill \begin{array}{c}{\left(\tilde{A}\left(\eta \left(t\right)\right)+\left({\gamma }^{-2}{\tilde{B}}_1\left(\eta \left(t\right)\right){\tilde{B}}_1^{\top }\left(\eta \left(t\right)\right)-{\tilde{B}}_2\left(\eta \left(t\right)\right){\tilde{B}}_2^{\top }\left(\eta \left(t\right)\right)\right)\tilde{X}\left(\eta \left(t\right)\right)\right)}^{\top }\tilde{S}\left(i\right)\\ +\tilde{S}\left(i\right)\left(\tilde{A}\left(\eta \left(t\right)\right)+\left({\gamma }^{-2}{\tilde{B}}_1\left(\eta \left(t\right)\right){\tilde{B}}_1^{\top }\left(\eta \left(t\right)\right)-{\tilde{B}}_2\left(\eta \left(t\right)\right){\tilde{B}}_2^{\top }\left(\eta \left(t\right)\right)\right)\tilde{X}\left(\eta \left(t\right)\right)\right)\\ +\sum _{j=1}^d{\pi }_{ij}\tilde{S}\left(j\right)<0,i=1,\dots ,d.\end{array}\end{array}$$

(23)

Since $\left(X_1\left(1\right),\dots ,X_1\left(d\right)\right)$ is the stabilising solution of the Riccati-type system (20) it follows that the stochastic system

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=\left(A\left(\eta \left(t\right)\right)+M\left(\eta \left(t\right)\right)X_1\left(\eta \left(t\right)\right)\right)x\left(t\right),\end{array}$$

where one denoted $M\left(i\right):={\gamma }^{-2}{B}_1\left(i\right)B_1^{\top }\left(i\right)-B_2\left(i\right)B_2^{\top }\left(i\right)$, is ESMS and therefore there exist the positive definite matrices $S_1\left(i\right),i=1,\dots ,d$ such that

$$\begin{array}{c}\hfill {\left(A\left(i\right)+M\left(i\right)X_1\left(i\right)\right)}^{\top }{S}_1\left(i\right)+S_1\left(i\right)\left(A\left(i\right)+M\left(i\right)X_1\left(i\right)\right)+\sum _{j=1}^d{\pi }_{ij}{S}_1\left(j\right)<0.\end{array}$$

(24)

Similarly, based on the fact that $\left(X_2\left(1\right),\dots ,X_2\left(d\right)\right)$ is the stabilising solution of (21), there exist the positive definite matrices $S_2\left(i\right),i=1,\dots ,d$ such that

$$\begin{array}{c}\hfill \begin{array}{c}{\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+N{X}_2\left(i\right)\right)\right)}^{\top }{S}_2\left(i\right)\\ +S_2\left(i\right)\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+N{X}_2\left(i\right)\right)\right)+\sum _{j=1}^d{\pi }_{ij}{S}_2\left(j\right)<0,i=1,\dots ,d.\end{array}\end{array}$$

(25)

Define

$$\begin{array}{c}\hfill \tilde{S}\left(i\right)=\left[\begin{array}{cccc}{S}_1\left(i\right)+(N-1)S_2\left(i\right)\hfill & -S_2\left(i\right)\hfill & \dots \hfill & -S_2\left(i\right)\hfill \\ -S_2\left(i\right)\hfill & S_1\left(i\right)+(N-1)S_2\left(i\right)\hfill & \dots \hfill & -S_2\left(i\right)\hfill \\ ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill \\ -S_2\left(i\right)\hfill & -S_2\left(i\right)\hfill & \dots \hfill & S_1\left(i\right)+(N-1)S_2\left(i\right)\hfill \end{array}\right]\end{array}$$

for which the inequalities (23) become:

$$\begin{array}{c}\hfill \mathcal{P}\left(i\right):=\left[\begin{array}{cccc}{\mathcal{P}}_1\left(i\right)\hfill & {\mathcal{P}}_2\left(i\right)\hfill & \dots \hfill & {\mathcal{P}}_2\left(i\right)\hfill \\ {\mathcal{P}}_2\left(i\right)\hfill & {\mathcal{P}}_1\left(i\right)\hfill & \dots \hfill & {\mathcal{P}}_2\left(i\right)\hfill \\ ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill \\ {\mathcal{P}}_2\left(i\right)\hfill & {\mathcal{P}}_2\left(i\right)\hfill & \dots \hfill & {\mathcal{P}}_1\left(i\right)\hfill \end{array}\right]<0,i=1,\dots ,d\end{array}$$

(26)

where

$$\begin{array}{c}\hfill \begin{array}{c}{\mathcal{P}}_1\left(i\right)={\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+(N-1)X_2\left(i\right)\right)\right)}^{\top }\left(S_1\left(i\right)+(N-1)S_2\left(i\right)\right)\\ +\left(S_1\left(i\right)+(N-1)S_2\left(i\right)\right)\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+(N-1)X_2\left(i\right)\right)\right)\\ +(N-1)\left(X_2\left(i\right)M\left(i\right)S_2\left(i\right)+S_2\left(i\right)M\left(i\right)X_2\left(i\right)\right)+\sum _{j=1}^d{\pi }_{ij}\left(S_1\left(j\right)+(N-1)S_2\left(j\right)\right)\end{array}\end{array}$$

and

$$\begin{array}{c}\hfill \begin{array}{c}{\mathcal{P}}_2\left(i\right)=-{\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+(N-1)X_2\left(i\right)\right)\right)}^{\top }{S}_2\left(i\right)\\ -S_2\left(i\right)\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+(N-1)X_2\left(i\right)\right)\right)\\ -\left(S_1\left(i\right)+S_2\left(i\right)\right)M\left(i\right)X_2\left(i\right)-X_2\left(i\right)M\left(i\right)\left(S_1\left(i\right)+S_2\left(i\right)\right)-\sum _{j=1}^d{\pi }_{ij}{S}_2\left(j\right),\end{array}\end{array}$$

$i=1,\dots ,d$. Defining the matrix

$$\begin{array}{c}\hfill \mathcal{T}=\left[\begin{array}{ccccc}{I}_n& I_n& I_n& \dots & I_n\\ 0& I_n& 0& \dots & 0\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \dots & I_n\end{array}\right]\end{array}$$

direct computations give

$$\begin{array}{c}\hfill \mathcal{T}\mathcal{P}\left(i\right){\mathcal{T}}^{-1}=\left[\begin{array}{cccc}{\mathcal{P}}_1\left(i\right)+(N-1){\mathcal{P}}_2\left(i\right)& 0& \dots & 0\\ ☆& {\mathcal{P}}_1\left(i\right)-{\mathcal{P}}_2\left(i\right)& \dots & 0\\ ☆& ☆& \ddots & 0\\ ☆& ☆& \dots & {\mathcal{P}}_1\left(i\right)-{\mathcal{P}}_2\left(i\right)\end{array}\right]\end{array}$$

in which ∗ denotes irrelevant elements. Due to the triangular structure of the above system it follows that the spectra of $\mathcal{P}\left(i\right),i=1,\dots ,d$ is given by the reunion of the spectra of ${\mathcal{P}}_1\left(i\right)+(N-1){\mathcal{P}}_2\left(i\right)$ and of ${\mathcal{P}}_1\left(i\right)-{\mathcal{P}}_2\left(i\right)$, respectively, $i=1,\dots ,d$. On the other hand, one may direcly check that ${\mathcal{P}}_1\left(i\right)+(N-1){\mathcal{P}}_2\left(i\right)$ coincide with the left hand sides of (24), $i=1,\dots ,d$. Further, using the above expressions of ${\mathcal{P}}_1\left(i\right)$ and ${\mathcal{P}}_2\left(i\right)$, it follows that

$$\begin{array}{c}\hfill \begin{array}{c}{\mathcal{P}}_1\left(i\right)-{\mathcal{P}}_2\left(i\right)={\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+N{X}_2\left(i\right)\right)\right)}^{\top }\left(S_1\left(i\right)+N{S}_2\left(i\right)\right)\\ +\left(S_1\left(i\right)+N{S}_2\left(i\right)\right)\left(A\left(i\right)+M\left(i\right)\left(X_1\left(i\right)+N{X}_2\left(i\right)\right)\right)+\sum _{j=1}^d{\pi }_{ij}\left(S_1\left(i\right)+N{S}_2\left(i\right)\right),\end{array}\end{array}$$

(27)

$i=1,\dots ,d$. Since (24) remains true if $S_1\left(i\right)$ are replaced by $ϵS_1\left(i\right),i=1,\dots ,d$ for any $ϵ>0$, from (25) it follows that for a small enough $ϵ>0$, ${\mathcal{P}}_1\left(i\right)-{\mathcal{P}}_2\left(i\right)<0$. Thus one concludes that $\mathcal{P}\left(i\right)<0,i=1,\dots ,d$ and therefore $\left(\tilde{X}\left(1\right),\dots ,\tilde{X}\left(d\right)\right)$ is the stabilising solution of the Riccati system (17). The fact that $\tilde{X}\left(i\right),i=1,\dots ,d$ given by (19) are positive semidefinite directly follows taking into account that

$$\begin{array}{c}\hfill \mathcal{T}\tilde{X}\left(i\right){\mathcal{T}}^{-1}=\left[\begin{array}{cccc}{X}_1\left(i\right)& 0& \dots & 0\\ ☆& X_1\left(i\right)+N{X}_2\left(i\right)& \dots & 0\\ ☆& ☆& \ddots & 0\\ ☆& ☆& \dots & X_1\left(i\right)+N{X}_2\left(i\right)\end{array}\right]\end{array}$$

and using the assumption that $X_1\left(i\right)$ and $X_2\left(i\right)$ are positive semidefinite. Thus the proof ends. □

Remark 3. 

Based on the expressions of $\tilde{X}\left(i\right)$ and $\tilde{F}\left(i\right),i=1,\dots ,d$, it follows that $\tilde{F}\left(i\right)$ has the following structure

$$\begin{array}{c}\hfill \tilde{F}\left(i\right)=\left[\begin{array}{cccc}{F}_1\left(i\right)& F_2\left(i\right)& \dots & F_2\left(i\right)\\ F_2\left(i\right)& F_1\left(i\right)& \dots & F_2\left(i\right)\\ ⋮& \ddots & ⋮& ⋮\\ F_2\left(i\right)& F_2\left(i\right)& \dots & F_1\left(i\right)\end{array}\right]\end{array}$$

(28)

where

$$\begin{array}{c}\hfill \begin{array}{ccc}{F}_1\left(i\right)\hfill & =& -B_2{\left(i\right)}^{\top }\left(X_1\left(i\right)+(N-1)X_2\left(i\right)\right)\hfill \\ F_2\left(i\right)\hfill & =& B_2{\left(i\right)}^{\top }{X}_2\left(i\right),i=1,\dots ,d.\hfill \end{array}\end{array}$$

(29)

4. The Data Packet Dropout Case

From (26) and (29) it follows that the control of each agent is determined as a combination of its own states and the states of all other network agents. The case when the states of all agents are available for every agent represents the nominal case and it will be denoted as the state $i=1$ of the set $\mathcal{D}$. In the case when the state of a certain agent is not available, the corresponding terms in the control expression of the other agents will be zero which is equivalent with the fact that ${\tilde{F}}_2\left(2\right)=0$. This case will be associated with the state $i=2$ of $\mathcal{D}$. Supposing that the communication network fails at a certain moment of time, it follows that $\tilde{F}\left(2\right)=diag\left(F_1\left(2\right),\dots ,F_1\left(2\right)\right)$ and $X_2\left(2\right)=0$. The condition $X_2\left(2\right)=0$ is accomplished if the weights corresponding to the coupling terms in (11) are taken to be zero, namely if $P\left(2\right)=0$ and if the corresponding row in the stationary transition rate matrix $\Pi$ has null elements. To conclude, in the above considered scenario in which either the communication network properly works or it completely fails leads to a Markovian model with $d=2$ states of the set $\mathcal{D}$. If the connection with a single agent, let say k is lost then the gain $\tilde{F}$ will have all extra diagonal elements of the k-th column equal to zero. Similarly, in the case when the connections with more agents fail, the columns of $\tilde{F}$ corresponding to these agents will have zero extra diagonal elements; in this situation the Markov chain will have maximum $2^N$ states.

In order to illustrate these ideas one considers a networked system with $N=100$ agents whose planar motions are described by the kinematic equations

$$\begin{array}{c}\hfill \begin{array}{ccc}{\ddot{x}}_k\left(t\right)\hfill & =& u_k\left(t\right)\hfill \\ {\ddot{y}}_k\left(t\right)\hfill & =& v_k\left(t\right),k=1,\dots ,N\hfill \end{array}\end{array}$$

where $x_k$ and $y_k$ denote the Cartesian coordinates and $u_i$ and $v_i$ their commanded accelerations, respectively. Two states of the Markov chain have been considered in this example. The first one corresponds to the case when the communication properly works and the states of all agents are available for all others; the second state of the Markov chain characterizes the situation in which each agent can only access its own state vector. Assuming that the quality outputs (denoted in Section 3 by $y_{1k}$) are ${\left[\begin{array}{cccc}{x}_k& y_k& u_k& v_k\end{array}\right]}^{\top }$ it follows that the matrix coefficients in the representation (1) are identical for both states of the Markov chain, namely

$$\begin{array}{c}A\left(i\right)=\left[\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 1\\ 0& 0& 0& 0\end{array}\right],B_1\left(i\right)=\left[\begin{array}{cc}1& 0\\ 0& 0\\ 0& 1\\ 0& 0\end{array}\right],B_2\left(i\right)=\left[\begin{array}{cc}0& 0\\ 1& 0\\ 0& 0\\ 0& 1\end{array}\right],\hfill \\ C\left(i\right)=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right],D\left(i\right)=\left[\begin{array}{cc}0& 0\\ 0& 0\\ 1& 0\\ 0& 1\end{array}\right],i=1,2,\hfill \end{array}$$

only the weights $Q_{k\ell }\left(i\right)$ in the cost function (11) being different for the two states. Thus for $i=1$ one chose $Q_{k\ell }=100·I_4$ (namely $P\left(1\right)={10}^2·I_4$) and for $i=2$, $Q_{k\ell }\left(2\right)=0_4$, for all $k\ne \ell$. It was assumed that stationary transition rate matrix of $\eta$ is

$$\begin{array}{c}\hfill \Pi =\left[\begin{array}{cc}-0.5& 0.5\\ 0& 0\end{array}\right].\end{array}$$

The elements of the second row of the stationary transition matrix were taken zero in order to accomplish the condition $X_2\left(2\right)=0$. Indeed, for $P\left(2\right)=0$ and for ${\pi }_{21}={\pi }_{22}=0$, the unique stabilising solution of the Riccati system (21) has the form $\left(X_2\left(1\right),0\right)$ since $\left(X_1\left(1\right),X_1\left(2\right)\right)$ is the stabilising solution of (20). The elements of the first row of $\Pi$ have been chosen such that the transition from the nominal first state to the second one corresponding to the network failure takes place in about 5 s as illustrated in Figure 1.

Solving the Riccati systems (20) and (21) for $\gamma =100$ one obtains

$$\begin{array}{c}\hfill \begin{array}{c}{X}_1\left(1\right)=\left[\begin{array}{cccc}1.7686& 1.2373& 0& 0\\ 1.2373& 1.7379& 0& 0\\ 0& 0& 1.7686& 1.2373\\ 0& 0& 1.2373& 1.7379\end{array}\right],\hfill \\ X_2\left(1\right)=\left[\begin{array}{cccc}204.1418& 2.0204& 0& 0\\ 2.0204& 1.0006& 0& 0\\ 0& 0& 204.1418& 2.0204\\ 0& 0& 2.0204& 1.0006\end{array}\right],\hfill \\ X_1\left(2\right)=\left[\begin{array}{cccc}1.4147& 1.0002& 0& 0\\ 1.0002& 1.4145& 0& 0\\ 0& 0& 1.4147& 1.0002\\ 0& 0& 1.0002& 1.4145\end{array}\right],X_2\left(2\right)=0_4\hfill \end{array}\end{array}$$

resulting the following gains

$$\begin{array}{c}\hfill \begin{array}{c}{F}_1\left(1\right)=\left[\begin{array}{cccc}-20.1631& -10.0793& 0& 0\\ 0& 0& -20.1631& -10.0793\end{array}\right],\hfill \\ F_2\left(1\right)=\left[\begin{array}{cccc}0.2024& 0.1001& 0& 0\\ 0& 0& 0.2024& 0.1001\end{array}\right],\hfill \\ F_1\left(2\right)=\left[\begin{array}{cccc}-0.1012& -0.1431& 0& 0\\ 0& 0& -0.1012& -0.1431\end{array}\right],F_2\left(2\right)=0_{2\times 4}.\hfill \end{array}\end{array}$$

Using the aforementioned gains one determined the agents planar trajectories for both cases when the network communication properly works and the case when it fails, respectively. Figure 2 presents two snapshots at t = 0.5 s and at t = 2 s obtained for random initial positions of the agents in both cases. One can see from the two snapshots in the upper half of the figure, that the matrix gain P giving the coupling between agents is important in determining a short settling time. By contrast, numerical simulations show that in the case when the communication network is not available the settling time is about 6.5 s.

5. Concluding Remarks

An optimal $H_{\infty }$ type control method for large-scale multi-agent systems with identical dynamics and independently actuated is presented. It is shown that regardless the number of agents, the optimal solution may be obtained solving two systems of algebraic Riccati equations whose dimension correspond to a single agent. Convergent iterative numerical procedures are presented and used for a case study revealing the benefits of the coupling between agents. The proposed design methodology may be also used in applications in which only some of the links between agents fail. The dimension of the Riccati equations remains the same but their number increases due to the larger number of states of the Markov chain. Future research will be devoted to applications in which a sensitivity analysis with respect to the transition probabilities will be included.