Transferable Utility Cooperative Differential Games with Continuous Updating Using Pontryagin Maximum Principle

Abstract

We consider a class of cooperative differential games with continuous updating making use of the Pontryagin maximum principle. It is assumed that at each moment, players have or use information about the game structure defined in a closed time interval of a fixed duration. Over time, information about the game structure will be updated. The subject of the current paper is to construct players’ cooperative strategies, their cooperative trajectory, the characteristic function, and the cooperative solution for this class of differential games with continuous updating, particularly by using Pontryagin’s maximum principle as the optimality conditions. In order to demonstrate this method’s novelty, we propose to compare cooperative strategies, trajectories, characteristic functions, and corresponding Shapley values for a classic (initial) differential game and a differential game with continuous updating. Our approach provides a means of more profound modeling of conflict controlled processes. In a particular example, we demonstrate that players’ behavior is braver at the beginning of the game with continuous updating because they lack the information for the whole game, and they are “intrinsically time-inconsistent”. In contrast, in the initial model, the players are more cautious, which implies they dare not emit too much pollution at first.

1. Introduction

Dynamic or differential games are an important subsection of game theory that investigates interactive decision-making over time. A differential game is when the evolution of the decision process takes place over a continuous time frame, and it generally involves a differential equation. Differential games provide an effective tool for studying a wide range of dynamic processes such as, for example, problems associated with controlling pollution where it is important to analyze the interactions between participants’ strategic behaviors and the dynamic evolution in the levels of pollutants that polluters release. In Carlson and Leitmann [1], a direct methodfor finding open-loop Nash equilibria for a class of differential n-player games is presented.

Cooperative optimization points to the possibility of socially optimal and individually rational solutions to decision-making problems involving strategic action over time. The approach to solving a static cooperative differential game is typically done using a two-step procedure. First, one determines the collective optimal solution and then payoffs are transferred and distributed by using one of the many accessible cooperative game solutions, such as core, Shapley value, nucleolus. In the dynamic cooperative game, it must be assured that, over time, all players will comply with the agreement. This will occur if each player’s profits in the cooperative situation at any intermediate moment dominate their non-cooperative profits. This property is known as time consistency and was introduced by Petrosjan (originally 1977) [2].

In order to derive equilibrium solutions, existing differential games often depend on the assumption of time-invariant game structures. However, the future is full of essentially unknown events. Therefore, it is necessary to consider that the information available to players about the future be limited. In the realm of dynamic updating, the Looking Forward Approach is used in game theory, and in differential games especially. The Looking Forward Approach solves the problem of modeling players’ behavior when the process information is dynamically updating. This means that the Looking Forward Approach does not use a target trajectory, but composes how a trajectory is to be used by players and how a cooperative payoff is to be allocated along that trajectory. The Looking Forward Approach was first presented in [3]. Afterward, the works in [4,5,6,7,8,9] were published.

In [10,11,12,13,14,15], a class of non-cooperative differential games with continuous updating is considered, and it is assumed that the updating process continues to develop over time. In the paper [10], the Hamilton–Jacobi–Bellman equations of the Nash equilibrium in the game with the continuous updating are derived. The work in [11] is devoted to the class of cooperative differential games with the transferable utility using Hamilton–Jacobi–Bellman equations, construction of characteristic function with continuous updating and several related theorems. Another result related to Hamilton–Jacobi–Bellman equations with continuous updating is devoted to the class of cooperative differential games with nontransferable utility [15]. The works in [13,14] are devoted to the class of linear-quadratic differential games with continuous updating. There the cooperative and non-cooperative cases are considered and corresponding solutions are obtained. In the paper [12], the explicit form of the Nash equilibrium for the differential game with continuous updating is derived by using the Pontryagin maximum principle. In this paper, the class of cooperative game models is examined and results concerning a cooperative setting such as the construction of the notion of cooperative strategies, the characteristic function, and cooperative solution for a class of games with continuous updating using Pontryagin maximum principle are presented. Theoretical results for three players are illustrated on a classic differential game model of pollution control presented in [16]. Another potentially important application of continuous updating approach is related to the class of inverse optimal control problems with continuous updating [17]. The approach can be used for human behavior modeling in engineering systems.

The class of differential games with dynamic and continuous updating has some similarities with Model Predictive Control (MPC) theory which is worked out within the framework of numerical optimal control in the books [18,19,20,21]. The current action of control is realized by solving a limited level of open-loop optimal control problem at each sampling moment in the MPC method. For linear systems, there is an explicit solution [22,23]. However, in general, the MPC approach needs to solve several optimization problems. Another series of related papers corresponding to the stable control category is [24,25,26,27], in which similar methods are considered for the linear-quadratic optimal control problem category. However, the goals of the current paper and the paper on continuous updating methods are different: when the information about the game process is continuous updated over time, the player’s behavior can be modeled. In [28,29], a similar issue is considered and the authors investigate repeated games with sliding planning horizons.

The paper is structured as follows. Section 2 starts with the initial differential game model and cooperative differential model. Section 3 demonstrates the knowledge of a differential game with continuous updating and a cooperative differential game with continuous updating by using the Pontryagin maximum principle method and obtains the definition of a characteristic function with continuous updating. It also presents the results of the theoretical portion. Section 4 gives an example of pollution control based on continuous updating. The conclusion is drawn in Section 5.

2. Initial Differential Game Model

2.1. Preliminary Knowledge

Consider the differential game starting with the initial position $x_0$ and evolving on time interval $[t_0,T]$. The equations of the system’s dynamics have the form

$$\dot{x}(t)=f(t,x,u),x(t_0)=x_0,$$

(1)

where $x\in {\mathbb{R}}^l$ is a set of variables that characterizes the state of the dynamical system at any instant of time during the play of the game, $u=(u_1,\dots ,u_n),u_i\in U_i\subset \mathrm{comp}{\mathbb{R}}^k$, is the control of player i. We shall use the notation $U=U_1\times U_2\times \dots \times U_n$.

The existence, uniqueness, and continuability of solution $x(t)$ for any admissible measurable controls $u_1(·),\dots ,u_n(·)$ was dealt with by Tolwinski, Haurie, and Leitmann [30]:

• $f(·):\mathbb{R}\times {\mathbb{R}}^n\times U\to {\mathbb{R}}^n$ is continuous.
• There exists a positive constant k such that

  $$\forall t\in [t_0,T]and\forall u\in U$$

  $$\parallel f(t,x,u)\parallel \le k(1+\parallel x\parallel )$$
• $\forall R>0$, $\exists K_R>0$ such that

  $$\forall t\in [t_0,T]and\forall u\in U$$

  $$\parallel f(t,x,u)-f(t,x^{\prime },u)\parallel \le K_R\parallel x-x^{\prime }\parallel$$

  for all x and $x^{\prime }$ such that

  $$\parallel x\parallel \le Rand\parallel x^{\prime }\parallel \le R$$
• for any $t\in [t_0,T]$ and $x\in X$ set

  $$G(x^t)=\{f(t,x,u)|u\in U\}$$

  is a convex compact from $R^l$.

The payoff of player i is then defined as

$$K_i(x_0,t_0,T;u)=\underset{t_0}{\overset{T}{\int }}{g}^i[t,x(t),u]dt,i\in N,$$

where $g^i[t,x,u]$, $f(t,x,u)$ are the integrable functions, $x(t)$ is the solution of the Cauchy problem (1) with fixed open-loop controls $u(t)=(u_1(t),\dots ,$$u_n(t))$. The strategy profile $u(t)=(u_1(t),\dots ,$$u_n(t))$ is called admissible if the problem (1) has a unique and continuable solution.

Let us agree that in a differential game, a subgame is a “truncated” version of the whole game. A subgame is a game in its own right and a subgame starts out at time instant $t\in [t_0,T]$, after a particular history of actions $u(t)$. Denote such a subgame by $\Gamma (x,t,T)$. (A remark on this notation is in order. Let the state of the game be defined by the pair $(x,t)$ and denote by $\Gamma (x,t,T)$ the subgame starting at date t with the state variable x; here, the model considered is of a finite horizon and we will have the terminal time T. If we take account of the infinite horizon, of course, one expects all corresponding value functions to depend on state and not on time.)

For each $(x,t,T)\in X\times [t_0,T]\times \mathbb{R}$, we define a subgame $\Gamma (x,t,T)$ by replacing the objective functional for player i and the system dynamics by

$$K_i(x,t,T;u(s))=\underset{t}{\overset{T}{\int }}{g}^i[s,x(s),u(s)]ds,i\in N,$$

and

$$\dot{x}(s)=f(s,x(s),u(s)),x(t)=x,$$

respectively. Therefore, $\Gamma (x,t,T)$ is a differential game defined on the time interval $[t,T]$ with initial condition $x(t)=x$.

2.2. Cooperative Differential Game Model

We adopt a cooperative game methodology to solve the differential game model with transferable utility. The steps are as follows.

• Define the cooperative behavior or strategies and corresponding cooperative trajectory.
• Determine the computation of the characteristic function values.
• Allocate among players a total cooperative payoff, such as the allocation belongs to the kernel, the bargaining set, the stable set, the core, the Shapley value and the nucleolus (see, e.g., Osborne and Rubinstein [31] for an introduction to these concepts).

First of all, we introduce the notions of cooperative strategies for players $u^{*}=(u_i^{*},\cdots ,u_n^{*})$ and the corresponding trajectory $x^{*}(t)$. Strategies $u^{*}(t)$ are called optimal strategies, i.e., a set of controls that maximizes the joint payoff of players:

$$u^{*}=(u_i^{*},\cdots ,u_n^{*})=arg\underset{u_1,\cdots ,u_n}{max}{\displaystyle \sum _{i\in N}}{K}_i(x_0,t_0,T;u).$$

(2)

Suppose that the maximum in (2) is achieved on the set of admissible strategies. If we substitute $u^{*}(t)$ into Equation (1), we can get the cooperative trajectory $x^{*}(t)$.

Consequently, to determine the way to distribute the maximum total payoff among players, it is fundamental to define the concept of the characteristic function of the coalition $S\subseteq N$. This characteristic function shows the strength of the coalition; remarkably, it allows us to take into account the players’ contributions to each coalition.

To define a cooperative game, a characteristic function must be introduced. We call $V(S;x_0,t_0,T)$, $S\subset N$ is a characteristic function for the initial differential game $\Gamma (x_0,t_0,T)$. Through a characteristic function we understand a map from the set of all possible coalitions:

$$\begin{array}{c}\hfill V(·):2^N\to \mathbb{R},\\ \hfill V(\varnothing )=0,\end{array}$$

which assigns to each coalition S the total payoff value which the players from S can guarantee when acting independently. An important property is the superadditivity of a characteristic function:

$$\begin{array}{c}V(S_1\cup S_2;x_0,t_0,T)\ge V(S_1;x_0,t_0,T)+V(S_2;x_0,t_0,T),\hfill \\ \forall S_1,S_2\subseteq N,S_1\cap S_2=\varnothing .\hfill \end{array}$$

(3)

The question of constructing a characteristic function is one of the main questions in cooperative game theory. Originally, the value of the characteristic function $V(S)$ was interpreted by von Neumann and Morgenstern (1944) as the maximum guaranteed payoff of coalition S that it can gain acting independently of other players [32]. Presently, it is known that there are various means of constructing characteristic functions in cooperative games, such as $\alpha$—c.f. [33], $\beta$—c.f. [34], $\zeta$—c.f. [35], and $\gamma$—c.f. [36].

Similar to the above, for each $(x^{*}(t),t,T)\in X\times [t_0,T]\times \mathbb{R}$, we define a cooperative subgame ${\Gamma }^c(x^{*}(t),t,T)$ (The superscript “c” means “cooperative”) along the cooperative trajectory $x^{*}(t)$ by replacing the objective functional for player i and the system dynamics by

$$\sum _{i\in N}{K}_i(x^{*}(t),t,T;u)=\sum _{i\in N}\underset{t}{\overset{T}{\int }}{g}^i[s,x(s),u(s)]ds,$$

and

$$\dot{x}(s)=f(s,x(s),u(s)),x(t)=x^{*}(t),$$

respectively. Therefore, ${\Gamma }^c(x^{*}(t),t,T)$ is a cooperative differential game defined on the time interval $[t,T]$ with initial condition $x(t)=x^{*}(t)$.

In this paper, we will adopt the constructive approach proposed by Petrosjan L. and Zaccour G. [37] with respect to putting together a $\delta$-characteristic function. $V(S;x^{*}(t),t,T)$ denotes the strength of a coalition S for the subgame $\Gamma (x^{*}(t),t,T)$, it can be calculated in two stages: in the beginning, we are obliged to compute the Nash equilibrium strategies $\left\{u_i^{NE}\right\}$ for all players $i\in N$, and second, we refrigerate the strategy for the players of $N\backslash S$, and the players of the coalition S seek to maximize their joint revenue ${\displaystyle \sum _{i\in S}}{K}_i$ on $u_S={\left\{u_i\right\}}_{i\in S}$. Thus, the definition of the characteristic function is given:

$$\begin{array}{ccc}\hfill V(S;x^{*}(t),t,T)=& & \left\{\begin{array}{cc}\underset{u_1,\cdots ,u_n}{max}{\displaystyle \sum _{i\in N}}{K}_i(x^{*}(t),t,T;u),\hfill & S=N,\hfill \\ \underset{\begin{array}{c}{u}_i,i\in S\end{array}}{max}{\displaystyle \sum _{i\in S}}{K}_i(x^{*}(t),t,T;u_S,u_{N\backslash S}^{NE}),\hfill & S\subset N,\hfill \\ 0,\hfill & S=\varnothing .\hfill \end{array}\right.\hfill \end{array}$$

Denote by $L(x^{*}(t),t,T)$ the set of imputations in the game $\Gamma (x^{*}(t),t,T)$:

$$\begin{array}{c}L(x^{*}(t),t,T)=\{\xi (x^{*}(t),t,T)=({\xi }_1(x^{*}(t),t,T),\dots ,{\xi }_n(x^{*}(t),t,T)):\hfill \\ \sum _{i=1}^n{\xi }_i(x^{*}(t),t,T)=V(N;x^{*}(t),t,T),\\ \hfill {\xi }_i(x^{*}(t),t,T)\ge V(\left\{i\right\};x^{*}(t),t,T),i\in N\},\end{array}$$

where $V(\left\{i\right\};x^{*}(t),t,T)$ is a value of characteristic function $V(S;x^{*}(t),t,T)$ for coalition $S=\left\{i\right\}$.

By $M(x^{*}(t),t,T)$ represent any cooperative solution or subset of imputation set $L(x^{*}(t),t,T)$:

$$M(x^{*}(t),t,T)\subseteq L(x^{*}(t),t,T).$$

In fact, the two extensively used cooperative solutions are the Shapley value and the core. In the following, we will consider a specific cooperative solution referred to as the Shapley value [38]. The Shapley value selects a single imputation, an n-vector denoted $sh(·)=(s{h}_1(·),s{h}_2(·),\cdots ,s{h}_n(·))$, satisfying three axioms: fairness, which means similar players are treated equally; efficiency (${\sum }_{i=1}^{n}s{h}_i(·)=V(N;·)$); and linearity (a relatively technical axiom required to obtain uniqueness). The Shapley value is defined in a unique way and is particularly suitable for a range of applications.

The Shapley value $sh(x^{*}(t),t,T)=(s{h}_1(x^{*}(t),t,T),\cdots ,s{h}_n(x^{*}(t),t,T))\in M(x^{*}(t),t,T)$ in the game ${\Gamma }^c(x^{*}(t),t,T)$ is a vector, such that

$$s{h}_i(x^{*}(t),t,T)=\sum _{K\subseteq N,i\in K}\frac{(k-1)!(n-k)!}{n!}[V(K;x^{*}(t),t,T)-V(K\backslash i;x^{*}(t),t,T)].$$

3. Differential Game with Continuous Updating

3.1. Preliminary Knowledge

In order to compose the corresponding differential game with continuous updating, we will apply the classic differential game with the specified duration of $\overline{T}$ to the continuously updated differential game. Consider the family of games $\Gamma (x,t,t+\overline{T})$ starting from the state x at an arbitrary time $t>t_0$. Furthermore, assume that the evolution of the state of the game $\Gamma (x,t,t+\overline{T})$ can be described by the ordinary differential equation

$$\begin{array}{c}{\dot{x}}_t(s)=f(s,x_t(s),u(t,s)),x_t(t)=x,\hfill \end{array}$$

(4)

where ${\dot{x}}_t(s)$ is the derivative with respect to s, $x_t\in {\mathbb{R}}^l$ are the state variables of a game that initials from time t, and $u(t,s)=(u_1(t,s),\dots ,u_n(t,s)),u_i(t,s)\in U_i\subset \mathrm{comp}{\mathbb{R}}^k,s\in [t,t+\overline{T}]$, indicates the control profile of the game that initials from time t at the instant time s.

For the game $\Gamma (x,t,t+\overline{T})$, the player’s payoff function has the following form,

$$K_i^t(x,t,t+\overline{T};u(t,s))=\underset{t}{\overset{t+\overline{T}}{\int }}{g}^i[s,x_t,u]ds,i\in N,$$

(5)

where $x_t(s)$, $u(t,s)$ are trajectory and strategy profile in the game $\Gamma (x,t,t+\overline{T})$.

The continuously updated differential games can be established in consonance with the following rules.

The instant time $t\in [t_0,+\infty )$ is continuously evolving, and appropriately, players continue to attain new information about the equations of motion and payment functions in the game $\Gamma (x,t,t+\overline{T})$.

The strategy vector $u(t)$ in the continuously updated differential game is as follows,

$${u(t)=u(t,s)|}_{s=t},t\in [t_0,+\infty ),$$

(6)

where $u(t,s)$, $s\in [t,t+\overline{T}]$ are strategies in the game $\Gamma (x,t,t+\overline{T})$.

Determine the trajectory $x(t)$ in the continuously updated differential game according to

$$\begin{array}{c}\dot{x}(t)=f(t,x,u),\hfill \\ x(t_0)=x_0,\hfill \\ x\in {\mathbb{R}}^l,\hfill \end{array}$$

(7)

where $u=u(t)$ are strategies in the continuously updated differential game (6), and $\dot{x}(t)$ is the derivative with respect to t. We assume that the strategy in the continuously updated differential game achieved using (6) is either admissible, or that the uniqueness and continuity of the solution of problem (7) can be guaranteed. The existence, uniqueness, and continuity conditions of the open-loop Nash equilibrium for the continuously updated differential game have been mentioned previously.

There is the indispensable difference between a continuously updated differential game and a classic differential game with the specified duration $\Gamma (x_0,t_0,T)$. In the case of classic game, the players are conducted by payoffs that they will finally gain within the time interval $[t_0,T]$; but in the game with continuous updating, they orient themselves toward the expected payoffs (5) at each time instant $t\in [t_0,T]$, which are computed due to the information determined by the interval $[t,t+\overline{T}]$, or the information that they possess at the instant time t. The subgame of the initial game has the form $\Gamma (x,t,T)$, by using the same method we can define a family subgames of the differential game with continuous updating at each t as $\Gamma (x_{t,s},s,t+\overline{T})$, where $x_{t,s}$ is the state at the instant time $s\in [t,t+\overline{T}]$. We will define this next.

First, we introduce the dynamic of the state:

$${\dot{x}}_t(\tau )=f(\tau ,x_t(\tau ),u(t,\tau )),x_t(s)=x_{t,s}.$$

(8)

Therefore, the payoff function of player i in a subgame with continuous updating $\Gamma (x_{t,s},s,t+\overline{T})$ has the form

$$K_i^t(x_{t,s},s,t+\overline{T};u)=\underset{s}{\overset{t+\overline{T}}{\int }}{g}^i[\tau ,x_t(\tau ),u(t,\tau )]d\tau ,i\in N,$$

where $x_t(\tau )$ satisfy (8) and $u(t,\tau )$, $\tau \in [s,t+\overline{T}]$, are strategies in the subgame $\Gamma (x_{t,s},s,t+\overline{T})$.

3.2. Cooperative Differential Game with Continuous Updating

In a cooperative setting, before starting the game, all players agree to behave jointly in an optimal way (cooperate).

3.2.1. The Approach to Define the Characteristic Function on the Interval $[s,t+\overline{T}]$

We introduce the notion of characteristic function ${\tilde{V}}_t(S;x,s,t+\overline{T})$, $\forall S\subseteq N$ defined for each subgame $\Gamma (x_{t,s},s,t+\overline{T})$, which $s\in [t,t+\overline{T}]$, $t\in [t_0,+\infty )$. Before introducing the characteristic function for the subgame $\Gamma (x_{t,s},s,t+\overline{T})$, it should be mentioned that from the Equation (4), we can derive that $x_{t,s}$ depends on the initial point x. Therefore, we can replace $x_{t,s}$ by x in the previous statement, such as by using $\Gamma (x,s,t+\overline{T})$ and $K_i^t(x,s,t+\overline{T};u)$ to represent the subgame and the payoff function for each player i of the subgame, respectively. Thus, the characteristic function is given:

$$\begin{array}{ccc}\hfill {\tilde{V}}_t(S;x,s,t+\overline{T})=& & \left\{\begin{array}{cc}\underset{u_1,\cdots ,u_n}{max}{\sum }_{i=1}^n{K}_i^t(x,s,t+\overline{T};u_1,\dots ,u_n),\hfill & S=N,\hfill \\ \underset{\begin{array}{c}{u}_i,i\in S\end{array}}{max}{\displaystyle \sum _{i\in S}}{K}_i^t(x,s,t+\overline{T};u_S,u_{N\backslash S}^{NE}),\hfill & S\subset N,\hfill \\ 0,\hfill & S=\varnothing ,\hfill \end{array}\right.\hfill \end{array}$$

where $x=x_t(t)$, $t\in [t_0,+\infty )$, $s\in [t,t+\overline{T}]$, which we have already described in (4). Moreover, $u_S={\left\{u_i\right\}}_{i\in S}$ is the strategy profile for the players in the coalition S.

We assume that superadditivity conditions for the characteristic function ${\tilde{V}}_t(S;x,s,t+\overline{T})$ are satisfied:

$$\begin{array}{ccc}\hfill & & {\tilde{V}}_t(S_1\cup S_2;x,s,t+\overline{T})\ge {\tilde{V}}_t(S_1;x,s,t+\overline{T})+{\tilde{V}}_t(S_2;x,s,t+\overline{T}),\hfill \\ & & \forall S_1,S_2\subseteq N,S_1\cap S_2=\varnothing .\hfill \end{array}$$

(9)

3.2.2. An Algorithm to Calculate Characteristic Function with Continuous Updating and the Shapley Value

The first three steps are to compute the necessary elements in order to define the characteristic function. In the next step, the Shapley value is computed.

Step 1: Optimizing the total payment of the grand coalition with continuous updating.

We shall refer to the cooperative differential game described above by ${\Gamma }^c(x,t,t+\overline{T})$, the duration of the game is $\overline{T}$. We believe that there are no inherent obstacles to cooperation between players, and their benefits can be transferred. More specifically, we assume that before the game actually starts, the players agree to cooperate in the game.

Definition 1.

Strategy profile ${\tilde{u}}^{*}(t,s)=({\tilde{u}}_1^{*}(t,s),\cdots ,{\tilde{u}}_n^{*}(t,s))$ is generalized open-loop cooperative strategies in a game with continuous updating if, for any fixed $t\in [t_0,+\infty )$ strategy profile, ${\tilde{u}}^{*}(t,s)$ are open-loop cooperative strategies in game ${\Gamma }^c(x,t,t+\overline{T})$.

Using the generalized open-loop cooperative strategies, it seems possible to define the solution concept for a game model with continuous updating.

Definition 2.

Strategy profile $u^{*}(t)=(u_1^{*}(t),\cdots ,u_n^{*}(t))$ are called open-loop cooperative strategies in a game with continuous updating when defined in the following way,

$$u^{*}(t)={\tilde{u}}^{*}{(t,s)|}_{s=t},t\in [t_0,+\infty ),$$

(10)

where ${\tilde{u}}^{*}(t,s)$ has defined the above.

We would like to interpret that the “intrinsically time-inconsistent” of players as follows:

• $u^{*}(t)$ in the moment t coincides with cooperative strategies in the game defined on the interval $[t,t+\overline{T}]$,
• $u^{*}(t+ϵ)$ in the instant $t+ϵ$ has to coincide with cooperative strategies in the game defined on the interval $[t+ϵ,t+ϵ+\overline{T}]$.

Trajectory $x^{*}(t)$ that corresponds to open-loop cooperative strategies with continuous updating $u^{*}(t)$ can be obtained from the system

$$\begin{array}{c}\dot{x}(t)=f(t,x,u^{*}),\hfill \\ x(t_0)=x_0,\hfill \\ x\in {\mathbb{R}}^l.\hfill \end{array}$$

Here, $x^{*}(t)$ denotes a cooperative trajectory with continuous updating.

Let there exist a set of controls

${\tilde{u}}^{*}(t,s)=({\tilde{u}}_1^{*}(t,s),\cdots ,{\tilde{u}}_n^{*}(t,s)),s\in [t,t+\overline{T}],t\in [t_0,+\infty )$

such that

$$\begin{array}{cc}\hfill & \underset{u_1,\cdots ,u_n}{max}\sum _{i=1}^n{K}_i^t(x^{*}(t),t,t+\overline{T};u_1,\cdots ,u_n)=\underset{u_1,\cdots ,u_n}{max}\sum _{i=1}^n\underset{t}{\overset{t+\overline{T}}{\int }}{g}^i[s,x_t(s),u(t,s)]ds\hfill \\ \hfill & s.t.x_t(s)satisfies{\dot{x}}_t(s)=f(s,x_t(s),u(t,s)),x_t(t)=x^{*}(t).\hfill \end{array}$$

(11)

The solution $x_t^{*}(s)$ of the system (11) corresponding to ${\tilde{u}}^{*}(t,s)$ is called the corresponding generalized cooperative trajectory.

Theorem 1.

Let $(i)f(s,·,u(t,s))$ be continuously differentiable at ${\mathbb{R}}^l$, $\forall s\in [t,t+\overline{T}]$,

$(ii)g^i(·,·,u(t,s))$ be continuously differentiable on $\mathbb{R}\times {\mathbb{R}}^l$,$\forall s\in [t,t+\overline{T}]$.

A set of strategies, ${\left\{{\tilde{u}}_i^{*}(t,s)\right\}}_{i\in N}$, provides generalized open-loop cooperative strategies in a differential game with continuous updating to the problem in (11), if for any fixed $t\in [t_0,T]$, there exists a costate variable ${\psi }^t(s)$ with $s\in [t,t+\overline{T}]$ so that the following relations are satisfied:

(1) ${\dot{x}}_t^{*}(s)=f(s,x_t^{*},{\tilde{u}}^{*})$, $x_t^{*}(t)=x^{*}(t),$ for $\forall s\in [t,t+\overline{T}]$,

(2) ${\tilde{u}}_i^{*}(t,s)=arg\underset{u_i\in U_i,i\in N}{max}{H}^t(s,x_t^{*}(s),u(t,s),{\psi }^t(s))$, where $s\in [t,t+\overline{T}]$, $i\in N$,

(3) ${\dot{\psi }}^t(s)=-\frac{\partial }{\partial x_t}{H}^t(s,x_t(s),{\tilde{u}}^{*}(t,s),{\psi }^t(s))$, where $s\in [t,t+\overline{T}]$,

${\psi }^t(t+\overline{T})=0$.

Remark 1.

Let fix $t\ge t_0$ and consider game ${\Gamma }^c(x,t,t+\overline{T})$.

The motion equation is in the form

$${\dot{x}}_t(s)=f(s,x_t(s),u(t,s)),x_t(t)=x^{*}(t),s\in [t,t+\overline{T}].$$

(12)

The payoff function of the grand coalition has the form

$$\sum _{i=1}^n{K}_i^t(x^{*}(t),t,t+\overline{T};u_1(t,s),\cdots ,u_n(t,s))=\sum _{i=1}^n{\int }_t^{t+\overline{T}}{g}^i[s,x_t(s),u(t,s)]ds.$$

(13)

For the optimization problem (12) and (13) Hamiltonian has the form

$$\begin{array}{cc}\hfill H^t(s,x_t(s),u(t,s),{\psi }^t(s))=& \sum _{i=1}^n{g}^i[s,x_t(s),u(t,s)]+{\psi }^t(s)f(s,x_t(s),u(t,s)),\hfill \\ \hfill & s\in [t,t+\overline{T}].\hfill \end{array}$$

If ${\tilde{u}}_i^{*}(t,s)$, $i\in N$ are the generalized open-loop cooperative strategies in the differential game with continuous updating; then, as stated in Definition 1, for every fixed $t\ge t_0$, ${\tilde{u}}_i^{*}(t,s)$, $i\in N$ is an open-loop cooperative strategy in game ${\Gamma }^c(x^{*}(t),t,t+\overline{T})$. Therefore, for any fixed $t\ge t_0$, the conditions (1)–(3) of the theorem are satisfied as necessary conditions for cooperative strategies in open-loop strategies (see in [39]).

On the other hand, if for every $t\ge t_0$, the Hamiltonian $H^t$ are concave in $(x_t,u(t,s))$, then the conditions of the theorem are sufficient for a cooperative open-loop solution [40].

Then, for any fixed $t\in [t_0,+\infty )$, we can obtain the generalized open-loop cooperative strategy ${\tilde{u}}^{*}(t,s)=({\tilde{u}}_i^{*}(t,s),\cdots ,{\tilde{u}}_n^{*}(t,s))$ and its corresponding state trajectory $x_t^{*}(s)$, $s\in [t,t+\overline{T}]$. Using Definitions 1 and 2, we can obtain the cooperative strategies with continuous updating ${\left\{u_i^{*}(t)\right\}}_{i\in N}$ and corresponding cooperative trajectory with continuous updating $x^{*}(t)$, $t\in [t_0,+\infty )$.

In order to get the characteristic function of the grand coalition in subgame $\Gamma (x^{*}(t),s,t+\overline{T})$, substituting ${\tilde{u}}^{*}(t,\tau )$ and $x_t^{*}(\tau )$ into the corresponding payoff function, denote ${\tilde{V}}_t(N;x^{*}(t),s,t+\overline{T})$ as the function of the coalition N. The current-value maximized cooperative payoff ${\tilde{V}}_t(N;x^{*}(t),s,t+\overline{T})$ can be expressed as

$${\tilde{V}}_t(N;x^{*}(t),s,t+\overline{T})=\sum _{i=1}^n\underset{s}{\overset{t+\overline{T}}{\int }}{g}^i[\tau ,x_t^{*}(\tau ),{\tilde{u}}^{*}(t,\tau )]d\tau .$$

Step 2: Computation of the generalized open-loop Nash equilibrium with continuous updating.

The problem of a non-cooperative subgame along the cooperative trajectory with continuous updating $\Gamma (x^{*}(t),s,t+\overline{T})$ can be stated as follows,

$$\begin{array}{cc}\hfill & \underset{u_i\in U_i}{max}{K}_i^t(x^{*}(t),s,t+\overline{T};u_i(t,s),{\tilde{u}}_{-i}^{NE}(t,s))=\underset{u_i\in U_i}{max}\underset{s}{\overset{t+\overline{T}}{\int }}{g}^i[\tau ,x_t(\tau ),u_i(t,\tau ),{\tilde{u}}_{-i}^{NE}(t,\tau )]d\tau \hfill \\ \hfill & s.t.x_t(\tau )satisfies(8)\hfill \end{array}$$

where ${\tilde{u}}_{-i}^{NE}(t,\tau )=({\tilde{u}}_1^{NE}(t,\tau ),\dots ,{\tilde{u}}_{i-1}^{NE}(t,\tau ),{\tilde{u}}_{i+1}^{NE}(t,\tau ),\dots ,{\tilde{u}}_n^{NE}(t,\tau ))$.

In this setting, the current-value Hamiltonian function can be written as

$$H_i^t(\tau ,x_t(\tau ),u(t,\tau ),{\psi }_i^t(\tau ))=g^i(\tau ,x_t,u)+{\psi }_i^t(\tau )f(\tau ,x_t,u),\tau \in [s,t+\overline{T}],i\in N,$$

where $\tau \in [s,t+\overline{T}]$, $t\in [t_0,+\infty )$. By using the Pontryagin maximum principle with continuous updating [12], we can get the open-loop Nash equilibrium ${\left\{{\tilde{u}}_i^{NE}(t,\tau )\right\}}_{i\in N}$, and the corresponding trajectory $x_t^{NE}(\tau )$, $\forall \tau \in [s,t+\overline{T}],t\in [t_0,+\infty )$. It is then easy to derive the characteristic function of a single-player coalition as follows, for each $i=1,2,\cdots ,n$

$${\tilde{V}}_t(\left\{i\right\};x^{*}(t),s,t+\overline{T})=\underset{s}{\overset{t+\overline{T}}{\int }}{g}^i[\tau ,x_t^{NE}(\tau ),{\tilde{u}}^{NE}(t,\tau )]d\tau ,\forall s\in [t,t+\overline{T}],t\in [t_0,+\infty ).$$

Step 3: Compute the characteristic function for all remaining possible coalitions with continuous updating.

Here, we need to compute only the coalitions that contain more than one player and exclude a grand coalition. There will be $2^n-n-2$ subsets obtained in the following way. We will apply the $\delta$-characteristic function so that players of S maximize their total payoff ${\displaystyle \sum _{i\in S}}{K}_i^t(x^{*}(t),s,t+\overline{T};u_S,{\tilde{u}}_{N\backslash S}^{NE})$ along the cooperative strategy with continuous updating $x^{*}(t)$, while the other players, those from $N\backslash S$, use generalized open-loop Nash-equilibrium strategies

$${\tilde{u}}_{N\backslash S}^{NE}={\left\{{\tilde{u}}_j^{NE}\right\}}_{j\in N\backslash S}.$$

Thus, we have a two-stage construction procedure for the characteristic function: (1) Find generalized open-loop Nash equilibrium strategies ${\tilde{u}}_i^{NE}(t,\tau )$ for all players $i\in N$, which we have found in the Step 2; (2) “Freeze” the Nash equilibrium strategies ${\tilde{u}}_j^{NE}(t,\tau )$ for players from $N\backslash S$, and, as for the player from the coalition S, maximize their total payoff over $u_S={\left\{u_i\right\}}_{i\in S}$. In order to compute the value function of the subgame $\Gamma (x^{*}(t),s,t+\overline{T})$, $\forall t\in [t_0,+\infty )$,$s\in [t,t+\overline{T}]$, we present the following concept.

Definition 3.

A set of strategies ${\tilde{u}}_S^{*}={\left\{{\tilde{u}}_i^{*}(t,\tau )\right\}}_{i\in S}$, $\tau \in [s,t+\overline{T}]$, provides a generalized open-loop optimal strategy for coalition $S\subset N$ in a subgame with continuous updating $\Gamma (x^{*}(t),s,t+\overline{T})$ when it is the solution obtained by using the Pontryagin maximum principle of the following problem

$$\begin{array}{cc}\hfill & \underset{u_S\in U_S}{max}\sum _{i\in S}{K}_i^t(x^{*}(t),s,t+\overline{T};u_S,{\tilde{u}}_{N\backslash S}^{NE})\hfill \\ \hfill & =\underset{u_S\in U_S}{max}\sum _{i\in S}\underset{s}{\overset{t+\overline{T}}{\int }}{g}^i[\tau ,x_t^S(\tau ),u_S(t,\tau ),{\tilde{u}}_{N\backslash S}^{NE}(t,\tau )]d\tau \hfill \\ \hfill & s.t.x_t^S(\tau )=f(\tau ,x_t^S(\tau ),u_S(t,\tau ),{\tilde{u}}_{N\backslash S}^{NE}(t,\tau )),x_t^S(s)=x_{t,s}.\hfill \end{array}$$

(14)

The Hamiltonian function of the problem (14) has the form, $\forall S$ (The uppercase letter “S” in the paper always denotes the coalition S, e.g., $x_t^S$, ${\psi }_S^t$, and $u_S)$$\subset N$:

$$\begin{array}{cc}\hfill H_S^t(\tau ,x_t^S(\tau ),u_S(t,\tau ),{\tilde{u}}_{N\backslash S}^{NE}(t,\tau ),{\psi }_S^t)=& \sum _{i\in S}{g}^i(\tau ,x_t^S,u_S,{\tilde{u}}_{N\backslash S}^{NE})\hfill \\ \hfill & +{\psi }_S^t(\tau )f(\tau ,x_t^S,u_S,{\tilde{u}}_{N\backslash S}^{NE}).\hfill \end{array}$$

Theorem 2.

Let

$(i)f(\tau ,·,u(t,\tau ))$ be continuously differentiable on ${\mathbb{R}}^l$, $\forall \tau \in [s,t+\overline{T}]$,

$(ii)g^i(·,·,u(t,\tau ))$ be continuously differentiable on $\mathbb{R}\times {\mathbb{R}}^l.$

A set of strategies, ${\tilde{u}}_S^{*}={\left\{{\tilde{u}}_i^{*}(t,\tau )\right\}}_{i\in S}$, provides a generalized open-loop optimal strategies of the coalition S in subgame with continuous updating $\Gamma (x^{*}(t),s,t+\overline{T})$ to the problem (14) if there exists $2^n-n-2$ costate functions ${\psi }_S^t(\tau )$, where $\tau \in [s,t+\overline{T}]$, $S\subset N$, so that, for $\forall s\in [t,t+\overline{T}]$, $t\in [t_0,+\infty )$, the following relations are satisfied:

(1) ${\dot{x}}_t^S(\tau )=f(\tau ,x_t^S,{\tilde{u}}_S^{*}(t,\tau ),{\tilde{u}}_{N\backslash S}^{NE})$, $x_t^S(s)=x_{t,s}$, for $\forall \tau \in [s,t+\overline{T}]$,

(2) ${\tilde{u}}_S^{*}(t,\tau )=arg\underset{u_S\in U_S}{max}{H}_S^t(x_t^S(\tau ),u_S(t,\tau ),{\tilde{u}}_{N\backslash S}^{NE}(t,\tau ),{\psi }_S^t(\tau ))$, for $\forall \tau \in [s,t+\overline{T}]$, where ${\tilde{u}}_S^{*}(t,\tau )={\left\{{\tilde{u}}_i^S(t,\tau )\right\}}_{i\in S}$

(3) ${\dot{\psi }}_S^t(\tau )=-\frac{\partial }{\partial x_t}{H}_S^t(\tau ,x_t(\tau ),{\tilde{u}}_S^{*}(t,\tau ),{\tilde{u}}_{N\backslash S}^{NE}(t,\tau ),{\psi }_S^t(\tau ))$, where $\tau \in [s,t+\overline{T}]$, $S\subset N$ ${\psi }_S^t(t+\overline{T})=0$, $S\subset N$.

Proof. 

Follow the proof of Theorem 1. □

Therefore, the agents in the coalition S will adopt the generalized open-loop optimal control ${\tilde{u}}_S^{*}(t,\tau )$ characterized in Theorem 2. Note that these controls are functions of fixed time $t\in [t_0,+\infty )$ and instant time $\tau \in [s,t+\overline{T}]$.

An illustration of the characteristic function for the coalition $S\subseteq N$ is provided in the following way,

$$\begin{array}{c}\hfill {\tilde{V}}_t(S;x^{*}(t),s,t+\overline{T})=\sum _{i\in S}\underset{s}{\overset{t+\overline{T}}{\int }}{g}^i[\tau ,x_t^S(\tau ),{\tilde{u}}_S^{*}(t,\tau ),{\tilde{u}}_{N\backslash S}^{NE}(t,\tau )]d\tau ,\\ \hfill \forall s\in [t,t+\overline{T}],t\in [t_0,+\infty ),\end{array}$$

where $x_t^S(\tau )$ is the trajectory at time instant $\tau \in [s,t+\overline{T}]$ when the players in coalition S use generalized open-loop optimal strategies ${\tilde{u}}_S^{*}(t,\tau )$, while players in $N\backslash S$ use generalized open-loop Nash equilibrium ${\tilde{u}}_{N\backslash S}^{NE}(t,\tau )$ that was already derived in Step 2.

For the characteristic function in the game model with continuous updating, first, suppose that the function ${\tilde{V}}_t(S;x^{*}(t),s,t+\overline{T})$, $\forall S\subseteq N$ can be continuously differentiated by $s\in [t,t+\overline{T}]$. Moreover, through $t\in [t_0,+\infty )$ can be integrated, the characteristic function in the game model with continuous updating $V(S;x^{*}(t),t,T)$ is defined as follows.

Definition 4.

Function $V(S;x^{*}(t),t,T)$, $t\in [t_0,T]$, $S\subseteq N$ is a characteristic function of the differential game with continuous updating $\Gamma (x^{*}(t),t,T)$, if it is defined as the following integral,

$$V(S;x^{*}(t),t,T)={\int }_t^T-\frac{d}{ds}{\tilde{V}}_{{\tau }^{\prime }}(S;x^{*}({\tau }^{\prime }),s,{\tau }^{\prime }+\overline{T}){|}_{s={\tau }^{\prime }}d{\tau }^{\prime },t\in [t_0,T],S\subseteq N,$$

(15)

where ${\tilde{V}}_{{\tau }^{\prime }}(S;x^{*}({\tau }^{\prime }),s,{\tau }^{\prime }+\overline{T})$, $s\in [{\tau }^{\prime },{\tau }^{\prime }+\overline{T}]$, ${\tau }^{\prime }\in [t,T]$, $S\subseteq N$ defined on the interval $[s,{\tau }^{\prime }+\overline{T}]$ is a characteristic function in game $\Gamma (x^{*}({\tau }^{\prime }),s,{\tau }^{\prime }+\overline{T})$.

In (15), we assume that the intergal is taken with a finite time interval because in this case we can only claim that the values of the characteristic function with continuous updating are finite. Later on in the example model, we shall calculate the characteristic function and Shapley value using the final interval method. We assume that superadditivity conditions are satisfied:

$$\begin{array}{ccc}\hfill & & V(S_1\cup S_2;x^{*}(t),t,T)\ge V(S_1;x^{*}(t),t,T)+V(S_2;x^{*}(t),t,T),\hfill \\ & & \forall S_1,S_2\subseteq N,S_1\cap S_2=\varnothing .\hfill \end{array}$$

(16)

Step 4: Compute the Shapley value based on the characteristic function with continuous updating.

Consider again the cooperative game model ${\Gamma }^c(x^{*}(t),t,T)$ with continuous updating. If the players are allowed to form different coalitions consisting of a subset of all players $K\subseteq N$. There are k players in the subset K. An imputation set of cooperative game ${\Gamma }^c(x^{*}(t),t,T)$ is the set $L(x^{*}(t),t,T)=\{\xi (x^{*}(t),t,T)=({\xi }_1(x^{*}(t),t,T),\cdots ,{\xi }_n(x^{*}(t),t,T))\}$, which satisfies the conditions

$$\begin{array}{cc}\hfill & {\xi }_i(x^{*}(t),t,T)\ge V(\left\{i\right\};x^{*}(t),t,T),i\in N;\hfill \\ \hfill & \sum _{i\in N}{\xi }_i(x^{*}(t),t,T)=V(N;x^{*}(t),t,T)\}.\hfill \end{array}$$

A cooperative solution or the optimal principle is a non-empty subset of the imputation set $L(x^{*}(t),t,T)$. In particular, the Shapley value $sh(x^{*}(t),t,T)=(s{h}_1(x^{*}(t),t,T),\cdots ,s{h}_n(x^{*}(t),t,T))$ is an imputation whose components are defined as

$$s{h}_i(x^{*}(t),t,T)=\sum _{K\subseteq N,i\in K}\frac{(k-1)!(n-k)!}{n!}[V(K;x^{*}(t),t,T)-V(K\backslash i;x^{*}(t),t,T)],$$

(17)

where $K\backslash i$ is the relative complement of i in K, the notion $V(K;x^{*}(t),t,T)$ is defined by Definition 4 and is the profit of coalition K. Meanwhile, $[V(K;x^{*}(t),t,T)-V(K\backslash i;x^{*}(t),t,T)]$ is the marginal contribution of player i to coalition K.

There are many other cooperative optimality principles, for example, the von Neumann–Morgenstern solution, N-core, and nucleus. In all cases they involve some subsets of the game imputation set.

4. A Cooperative Differential Game for Pollution Control

Let us consider the following game proposed by Long [41]. When countries are indexed by $i\in N$, we denote that $n=|N|$. It is assumed that each player has an industrial production site and the production is proportional to the pollutant $u_i$. Therefore, the player’s strategy is to decide the amount of pollutants emitted into the atmosphere.

4.1. Initial Game Model

Pollution accumulates over time. We denote by $x(t)$ the stock of pollution at time t and assume that the countries “contribute” to the same stock of pollution. For simplicity, the evolution of stock $x(t)$ is represented by the following linear equation:

$$\dot{x}(t)=\sum _{i=1}^n{u}_i(t)-\delta x(t),x(t_0)=x_0,$$

where $\delta$ is a constant rate of decay, in other words, the absorption rate of pollution by nature.

In the following, we assume that the absorption coefficient $\delta$ is equal to zero:

$$\dot{x}(t)=\sum _{i=1}^n{u}_i(t),x(t_0)=x_0.$$

(18)

Pollution is a “public bad” because it exerts adverse affects on health, quality of life, and productivity. We assume that these adverse effects can be represented by having x as an argument of the instantaneous social welfare function $F_i$, with negative derivative:

$$F_i=F_i(x,t,u_i),\frac{\partial F_i}{\partial x}<0.$$

In each country, aggregate social welfare is taken to be the integral of the instantaneous social welfare. Thus, the payoff of the player i can be formulated as follows,

$$K_i(x_0,t_0,T;u)={\int }_{t_0}^T{F}_i(x,t,u_i)dt.$$

For tractability, the function $F_i$ is often assumed to take the separable form:

$$F_i(x,t,u_i(t))=R_i(u_i(t))-D_i(x),$$

where $R_i(u_i)$ may be thought of as the utility of the benefit, and $D_i(x)$ as the “disutility” caused by pollution. Following standard practice, we take it that $R_i(u_i)$ is strictly concave and increasing in $u_i$, and that $D_i(x)$ is convex and increasing in x. The possibility that $D_i$ is linear is not ruled out.

We assume that the environmental damage cost of player i caused by the pollution stock is $D_i(x)=d_{i}x$ and the damage cost $D_i(x)$ increases convexly. In the environmental economics literature, the typical assumption is that the production income function of player i can be expressed as a function of emissions, namely, $R_i(u_i(t))=b_i{u}_i-\frac{1}{2}{u}_i^2$, satisfying $R_i(0)=0$, where $b_i$ and $d_i$ are positive parameters. For the above benefit function to have a concave increase in emissions, we impose the restriction $u_i(t)\in (0,b_i)$.

Suppose that the game is played in a cooperative scenario in which players have the opportunity to cooperate in order to achieve maximum total payoff:

$$\underset{u_1,u_2,...,u_n}{max}\sum _{i=1}^n{K}_i(x_0,t_0,T;u)=\sum _{i=1}^n{\int }_{t_0}^T((b_i-\frac{1}{2}{\tilde{u}}_i){\tilde{u}}_i-d_{i}x)dt$$

(19)

To solve the optimization problem in (19) and (18), we invoke the Pontryagin maximum principle to characterize the solution as follows. Obviously, these are linear state games (These are games for which the system dynamics and the utility functions are polynomials of degree 1 with respect to the state variables and which satisfy a certain property (described below) concerning the interaction between control variables and state variables. We call this class of games linear state games.). This shows that these games have the property that their open-loop Nash equilibrium are Markov perfect. The class of linear state games has a very useful property. The linearity in the state variables together with the decoupled structure between the state variables and the control variables implies that the open-loop equilibrium is Markov perfect and that the value functions are linear in the state variables.

It is obvious to demonstrate that the optimal emissions control of player i for an initial differential game model is given by

$${\tilde{u}}_i(t)=b_i-\sum _{i=1}^n{d}_i(T-t),i\in N.$$

(20)

To obtain the cooperative state trajectory for the initial differential game, it suffices to insert ${\tilde{u}}_i(t)$ in (20) into the dynamics and to solve the differential equation to get

$${\tilde{x}}^{*}(t)=x_0+(\sum _{i=1}^n{b}_i-n\sum _{i=1}^n{d}_{i}T)(t-t_0)+n\sum _{i=1}^n{d}_i\frac{t^2-t_0^2}{2}.$$

(21)

4.2. A Pollution Control Game Model with Continuous Updating

In the game $\Gamma (x,t,t+\overline{T})$, the dynamics of the total amount of pollution $x_t(s)$ is described by

$$\dot{x_t}(s)=\sum _{i=1}^n{u}_i(t,s),x_t(t)=x,$$

(22)

in which we assume that the absorption coefficient corresponding to the natural purification of the atmosphere is equal to zero.

The instantaneous payoff of $i\text{-}th$ player is defined as

$$R_i(u_i(t,s))=b_i{u}_i(t,s)-\frac{1}{2}{u}_i^2(t,s),i\in N.$$

Due to decontamination, each player is compelled to bear the cost. Therefore, the instantaneous utility of the $i\text{-}th$ player is equal to $R_i(u_i(t,s))-d_i{x}_t(s)$, where $d_i>0$.

Thus the payoff of the player i is defined as

$$K_i^t(x,t,t+\overline{T};u)={\int }_t^{t+\overline{T}}((b_i-\frac{1}{2}{u}_i)u_i-d_i{x}_t)ds,$$

where $u_i=u_i(t,s)$ is the control of the player i at the instant time $s\in [t,t+\overline{T}]$, $x_t=x_t(s)$ is the pollution accumulation at the same time s.

Therefore, the payoff function of the player i in the subgame with continuous updating $\Gamma (x,s,t+\overline{T})$ is given by

$$K_i^t(x,s,t+\overline{T};u)=\underset{s}{\overset{t+\overline{T}}{\int }}((b_i-\frac{1}{2}{u}_i(t,\tau ))u_i(t,\tau )-d_i{x}_t(\tau ))d\tau ,i\in N,$$

where $x_t(\tau )$, $u(t,\tau )$, and $\tau \in [s,t+\overline{T}]$ are both the trajectory and strategies in game $\Gamma (x,s,t+\overline{T})$. The dynamics of the state is given by

$${\dot{x}}_t(\tau )=\sum _{i=1}^n{u}_i(t,\tau ),x_t(s)=x_{t,s}.$$

(23)

Step 1: Optimizing the total payment of the grand coalition with continuous updating.

Consider the game in a cooperative form. This means that all players will work together to maximize their total payoff. We seek the optimal profile of strategies ${\tilde{u}}^{*}(t,s)=({\tilde{u}}_1^{*}(t,s),...,{\tilde{u}}_n^{*}(t,s))$ such that ${\sum }_{i=1}^n{K}_i^t\to \underset{u_1,u_2,...,u_n}{max}$.

The optimization problem is as follows,

$$\begin{array}{cc}\hfill & \sum _{i=1}^n{K}_i^t(x,t,t+\overline{T};u)=\sum _{i=1}^n{\int }_t^{t+\overline{T}}((b_i-\frac{1}{2}{u}_i(t,s))u_i(t,s)-d_i{x}_t(s))ds\to \underset{u_1,u_2,...,u_n}{max}\hfill \\ \hfill & s.t.x_t(s)satisfies(22).\hfill \end{array}$$

(24)

In order to deal with the problem (24), we use the classical Pontryagin maximum principle. The corresponding Hamiltonian is

$$H^t(s,x_t(s),u(t,s),{\psi }^t(s))=\sum _{i=1}^n(b_i-\frac{1}{2}{u}_i)u_i-\sum _{i=1}^n{d}_i{x}_t+{\psi }^t(s)(u_1+u_2+...+u_n).$$

The first order partial derivatives w.r.t. $u_i$’s are

$$\frac{\partial H^t}{\partial u_i}(s,x_t,u,{\psi }^t)=b_i-u_i+{\psi }^t=0,$$

and the Hessian matrix $\frac{{\partial }^2{H}^t}{\partial u^2}(s,x_t,u,{\psi }^t)$ is negative definite, all at once, we can conclude that the Hamiltonian $H^t$ is concave w.r.t. $u_i$. Here, we obtain the cooperative strategies:

$${\tilde{u}}_i^{*}(t,s)=b_i+{\psi }^t(s)$$

Considering the Pontryagin’s maximum principle, when dealing with the costate variable

$${\dot{\psi }}^t(s)=\sum _{i=1}^n{d}_i,{\psi }^t(t+\overline{T})=0$$

if we set ${\sum }_{i=1}^n{d}_i=d_N$, so, we can get ${\psi }^t(s)=-d_N(t+\overline{T}-s)$. Finally, the form of the cooperative strategies is

$${\tilde{u}}_i^{*}(t,s)=b_i-d_N(t+\overline{T}-s)$$

(25)

and from (22) we get the optimal (cooperative) trajectory:

$$x_t^{*}(s)=x+b_N(s-t)-n{d}_N(t+\overline{T})(s-t)+n{d}_N\frac{s^2-t^2}{2},$$

(26)

where $x=x_t(t)$, $d_N={\sum }_{i=1}^n{d}_i$, $b_N={\sum }_{i=1}^n{b}_i$.

According to the procedure (10), we construct open-loop optimal cooperative strategies with continuous updating:

$$u_i^{*}(t)={\tilde{u}}_i^{*}{(t,s)|}_{s=t}=b_i-d_N\overline{T}.$$

(27)

After substituting $u_i^{*}(t)$ into the differential Equation (18), we can arrive at the optimal cooperative trajectory $x^{*}(t)$ with continuous updating:

$$x^{*}(t)=x_0+b_N(t-t_0)-n{d}_N\overline{T}(t-t_0).$$

(28)

The results of the comparison of the cooperative strategies, corresponding trajectories between initial differential game model and the differential game with continuous updating obtained are graphically shown in Figure 1 and Figure 2.

From Figure 1 we can see that the optimal control with continuous updating is more stable than the optimal control in the initial game model. We can also see that, from the time $t=4$, the optimal control in the initial game is greater than it with continuous updating, which means players should increase the pollution emissions into the atmosphere in the initial differential game model, a harmful result. This occurs because in the initial game model, players have the whole information of the game on the interval $[t_0,T]$, players are more cautious, and they dare not emit too much pollution at first. However, in real life, it is impossible to have the information for the whole time interval. Therefore, we consider the game with continuous updating, at each time instant t, players have the information only on $[t,t+\overline{T}]$. In the case of continuous updating, the players are brave enough to emit more pollution because of lacking the information for the whole game.

We can see from Figure 2 that, starting from $t=0$ to $t=8$, the pollution accumulation in the initial game model is less than the model with continuous updating. Because in the initial game model, players are more knowledgeable, they know the information from the whole time interval, which leads to lower pollution accumulation because the players are cautious. Starting from time $t=8$, pollution with continuous updating is less than pollution in the initial game because the knowledge for players in the model with continuous updating is close to the initial game model as time goes on. Using the continuous updating method can help us to make our modeling more consistent with the actual situation.

Next, for a given subgame $\Gamma (x^{*}(t),s,t+\overline{T})$ of a differential game with continuous updating $\Gamma (x^{*}(t),t,t+\overline{T})$, the characteristic function for the grand coalition N is given by ${\tilde{V}}_t(N;x^{*}(t),s,t+\overline{T})$, which can be represented as

$${\tilde{V}}_t(N;x^{*}(t),s,t+\overline{T})=\sum _{i=1}^n{\int }_s^{t+\overline{T}}((b_i-\frac{1}{2}{\tilde{u}}_i^{*}(t,\tau )){\tilde{u}}_i^{*}-d_i{x}_t^{*}(\tau ))d\tau ,$$

where $x_t^{*}(\tau )$ satisfies (26) with $x=x^{*}(t)$, ${\tilde{u}}_i^{*}(t,\tau )$ satisfies (25). Therefore, we can get the value function of the grand coalition N

$$\begin{array}{cc}\hfill {\tilde{V}}_t(N;x^{*}(t),s,t+\overline{T})=& (t+\overline{T}-s)[\frac{1}{2}{\tilde{b}}_N-d_N{x}^{*}(t)+\frac{d_N{b}_N}{2}(t-\overline{T}-s)\hfill \\ & -\frac{1}{3}n{d}_N^2({(t+\overline{T}-s)}^2-\frac{3}{2}{\overline{T}}^2)],\hfill \end{array}$$

(29)

where $d_N={\sum }_{i=1}^n{d}_i$, $b_N={\sum }_{i=1}^n{b}_i$ has been defined above and ${\tilde{b}}_N={\sum }_{i=1}^n{b}_i^2$. Note that in (29), $x^{*}(t)$ represents the cooperative pollution with continuous updating at the time t.

For our problem, we can also use the dynamic programming method based on the Hamilton–Jacobi–Bellman equation. It is straightforward to verify the Bellman function of the form ${\tilde{V}}_t=A(t,s)x_t(s)+B(t,s)$, and we get the same result as Pontryagin maximum principle.

Step 2: The computation of the generalized open-loop Nash equilibrium with continuous updating.

The Hamiltonian for each player $i=1,2,\cdots ,n$ is

$$H_i^t(\tau ,x_t(\tau ),u(t,\tau ),{\psi }_i^t(\tau ))=(b_i-\frac{1}{2}{u}_i)u_i-\sum _{i=1}^n{d}_i{x}_t+{\psi }_i^t(\tau )(u_1+u_2+\cdots +u_n)$$

its first-order partial derivatives w.r.t. $u_i$’s are

$$\frac{\partial H_i^t}{\partial u_i}(\tau ,x_t(\tau ),u(t,\tau ),{\psi }_i^t(\tau ))=b_i-u_i+{\psi }_i^t=0,$$

and the Hessian matrix $\frac{{\partial }^2{H}_i^t}{\partial u_i^2}(\tau ,x_t,u,{\psi }_i^t)$ is the negative definite whence we conclude that the Hamiltonian $H_i^t$ is concave w.r.t. $u_i$. We obtain optimal controls

$${\tilde{u}}_i^{NE}(t,\tau )=b_i-d_i(t+\overline{T}-\tau ),i=1,2,\cdots ,n.$$

(30)

As for the subgame start at time instant $s\in [t,t+\overline{T}]$, we can easily derive the corresponding trajectory (for the Nash equilibrium case) of subgame $\Gamma (x^{*}(t),s,t+\overline{T})$ along the cooperative trajectory, in other words $x_{t,s}=x_t^{*}(s)$ is

$$\begin{array}{cc}\hfill x_t^{NE}(\tau )& =x_t^{*}(s)+b_N(\tau -s)+d_N(\frac{{(t+\overline{T}-\tau )}^2}{2}-\frac{{(t+\overline{T}-s)}^2}{2})\hfill \\ \hfill & =x^{*}(t)+b_N(\tau -t)-n{d}_N(t+\overline{T})(s-t)+n{d}_N\frac{s^2-t^2}{2}\hfill \\ \hfill & +\frac{d_N}{2}({(t+\overline{T}-\tau )}^2-{(t+\overline{T}-s)}^2).\hfill \end{array}$$

The maximum of the payoff for each player $i=1,2,\cdots ,n$ in the subgame starting from the time instant s and the state $x^{*}(t)$ has the form

$$\begin{array}{cc}\hfill {\tilde{V}}_t(\left\{i\right\};x^{*}(t),s,t+\overline{T})=& {\int }_s^{t+\overline{T}}((b_i-\frac{1}{2}{\tilde{u}}_i^{NE}(t,\tau )){\tilde{u}}_i^{NE}-d_i{x}_t^{NE}(\tau ))d\tau \hfill \\ \hfill & =(t+\overline{T}-s)[\frac{b_i^2}{2}-d_i{x}^{*}(t)+\frac{n{d}_i{d}_N}{2}(s-t)(t-s+2\overline{T})\hfill \\ \hfill & -\frac{1}{6}{d}_i^2{(t+\overline{T}-s)}^2+\frac{1}{3}{d}_i{d}_N{(t+\overline{T}-s)}^2-\frac{1}{2}{d}_i{b}_N(s+\overline{T}-t)].\hfill \end{array}$$

Step 3: The computation of the characteristic function for all remaining possible coalitions in differential games with continuous updating.

It is possible to calculate the controls and the corresponding value functions for different coalitions. Nonetheless, the form will depend on how we define their respective optimal control problems.

Let us build up the characteristic function hinged on the approach of $\delta$—c.f. The characteristic function of coalition S is calculated in two stages: the first stage is already done (we have already found the Nash equilibrium strategies for each player in Step 2); at the second stage, it is assumed that the remaining players $j\in N\backslash S$ carry out their Nash optimal strategies ${\tilde{u}}_j^{NE}(t,\tau )$ although the players from coalition S explore to make their joint payoff ${\displaystyle \sum _{i\in S}}{K}_i$ maximal. Consider the case of S-coalition. It seems constructive to perform calculations in detail. The respective Hamiltonian for the coalition S is

$$H_S^t(\tau ,x_t(\tau ),u(t,\tau ),{\psi }_S^t(\tau ))=\sum _{i\in S}((b_i-\frac{1}{2}{u}_i)u_i)-d_S{x}_t+{\psi }_S^t(\sum _{i\in S}{u}_i+\sum _{j\in N/S}{\tilde{u}}_j^{NE}),$$

where $d_S={\displaystyle \sum _{i\in S}}{d}_i$. Note that we substituted $u_j,j\in N/S$ by ${\tilde{u}}_j^{NE}$ which was found earlier.

The optimal strategies of players in coalition S are ${\tilde{u}}_S^{*}(t,\tau )$, which satisfies

$${\tilde{u}}_i^{*}(t,\tau )=b_i+{\psi }_S^t(\tau ),i\in S.$$

The differential equation for ${\psi }_S^t(\tau )$ is ${\dot{\psi }}_S^t(\tau )=d_S$ which is solved to ensure that ${\psi }_S^t(t+\overline{T})=0$. Eventually, we get ${\psi }_S^t(\tau )=-d_S(t+\overline{T}-\tau )$. We substitute the obtained expression for ${\psi }_S^t(\tau )$ into ${\tilde{u}}_i^{*}$, and then get

$${\tilde{u}}_i^{*}(t,\tau )=b_i-d_S(t+\overline{T}-\tau ),i\in S$$

where $d_S={\displaystyle \sum _{i\in S}}{d}_i$. We see that the player out of coalition S implements their optimal strategy ${\tilde{u}}_S^{*}$ while the left out players adhere to their Nash equilibrium.

In the next step, we integrate (23) start from the point $x_t^{*}(s)$ to get $x_t^S(\tau )$:

$$\begin{array}{cc}\hfill x_t^S(\tau )=& x_t^{*}(s)+b_N(\tau -s)+\frac{{(t+\overline{T}-\tau )}^2}{2}((k-1)d_S+d_N)\hfill \\ & -\frac{{(t+\overline{T}-s)}^2}{2}((k-1)d_S+d_N).\hfill \end{array}$$

$x_t^S(\tau )$ is the trajectory in the subgame $\Gamma (x^{*}(t),s,t+\overline{T})$ of the game $\Gamma (x^{*}(t),t,t+\overline{T})$ starting at time instant s at $x_t^{*}(s)$, when players from coalition S use strategies ${\tilde{u}}_S^{*}(t,\tau )$, and players from coalition $N\backslash S$ use ${\tilde{u}}^{NE}(t,\tau )$. If we consider the case taken along the cooperative trajectory, then we can substitute the $x_t^{*}(s)$ we have already obtained in (26) to get the state variable $x_t^S(\tau )$ that depends on $x=x^{*}(t)$.

$$\begin{array}{cc}\hfill x_t^S(\tau )=& x^{*}(t)+b_N(\tau -t)-n{d}_N(t+\overline{T})(s-t)+n{d}_N\frac{s^2-t^2}{2}+\hfill \\ & \frac{{(t+\overline{T}-\tau )}^2}{2}((k-1)d_S+d_N)-\frac{{(t+\overline{T}-s)}^2}{2}((k-1)d_S+d_N).\hfill \end{array}$$

The respective value of the characteristic function ${\tilde{V}}_t(S;x^{*}(t),s,t+\overline{T})$ is

$$\begin{array}{c}{\tilde{V}}_t(S;x^{*}(t),s,t+\overline{T})=(t+\overline{T}-s)[\frac{1}{2}{\tilde{b}}_S-d_S{x}^{*}(t)+\frac{d_S{d}_{N}n}{2}(t+2\overline{T}-s)(s-t)+\hfill \\ \hfill {(t+\overline{T}-s)}^2(\frac{k-2}{6}{d}_S^2+\frac{d_S{d}_N}{3})-\frac{d_S{b}_N}{2}(s+\overline{T}-t)].\end{array}$$

According to Definition 4, the characteristic function of a differential game with continuous updating has the following form,

$$\begin{array}{cc}\hfill V(S;x^{*}(t),t,T)=& \underset{t}{\overset{T}{\int }}-\frac{d{\tilde{V}}_{{\tau }^{\prime }}(S;x^{*}({\tau }^{\prime }),s,{\tau }^{\prime }+\overline{T})}{ds}{|}_{s={\tau }^{\prime }}d{\tau }^{\prime }\hfill \\ \hfill =& (T-t)[\frac{1}{2}{\tilde{b}}_S-d_S{x}_0+{\overline{T}}^2(\frac{k-2}{2}{d}_S^2+(1-n)d_S{d}_N)\hfill \\ \hfill & -\frac{d_S(b_N-n{d}_N\overline{T})}{2}(T+t-2t_0)]\hfill \end{array}$$

Check the superadditivity condition (16) for constructed characteristic function

$V(S;x^{*}(t),t,T)$. It turns out that for any $S,P\subseteq N$ and $S\cap P=\varnothing$, let $|S|=k\ge 1$,$|P|=m\ge 1$ the following holds,

$$\begin{array}{cc}\hfill & V(S\cup P;x^{*}(t),t,T)-V(S;x^{*}(t),t,T)-V(P;x^{*}(t),t,T)\hfill \\ \hfill & ={\overline{T}}^2[(k+m-2)d_S{d}_P+\frac{k}{2}{d}_P^2+\frac{m}{2}{d}_S^2]\ge 0.\hfill \end{array}$$

Thus, the $\delta$-characteristic function $V(S;x^{*}(t),t,T)$ is a superadditive function without any additional conditions applied to the parameters of the model.

In the following figure, we will compare the characteristic function for the grand coalition N between the initial game model and a differential game with continuous updating.

Figure 3 demonstrates the reason accounting for why the value of a characteristic function in the initial model is greater than that of continuous updating is that the complexity of the information within a continuous updating setting can reduce the effectiveness of the coalition. It should be noted that the continuous updating case is more realistic. We can conclude that the payoff of the coalition decreases because, as time goes on, pollution accumulates in the air. The player’s payoff depends on levels of pollution and payoff decreases as pollution increases. It should also be noted that the coalition’s effectiveness decreases in the initial game model at a faster rate than it does with continuous updating.

Step 4: Compute the Shapley value based on the characteristic function with continuous updating.

Any of the known principles of optimality can be applied to find a cooperative solution. First of all, the notion $\sum d_j{d}_l$ (Here we should note that $d_k{d}_j=d_j{d}_k.$) represents the interaction of cost among players. Now, consider the cooperative solution of a differential game with continuous updating. According to procedure (17), we construct the Shapley value for any $i\in N$ with continuous updating using the characteristic function with continuous updating of the auxiliary subgame and get

$$\begin{array}{cc}\hfill s{h}_i(x^{*}(t),t,T)=& \sum _{K\subseteq N,i\in K}\frac{(k-1)!(n-k)!}{n!}[V(K;x^{*}(t),t,T)-V(K\backslash i;x^{*}(t),t,T)]\hfill \\ \hfill =& (T-t)[-d_i{x}_0+\frac{1}{2}{b}_i^2+\frac{n{d}_i{d}_N\overline{T}-d_i{b}_N}{2}(T+t-2t_0)\hfill \\ \hfill & +{\overline{T}}^2(\frac{1-2n}{3}{d}_i{d}_N-\frac{4+n}{12}{d}_i^2+\frac{1}{3}(\sum _{\begin{array}{c}j,l\ne i\\ j\ne l\in N\end{array}}{d}_j{d}_l)+\frac{1}{4}{\tilde{d}}_N)].\hfill \end{array}$$

(31)

The graphic representation of the Shapley value for subgames with continuous updating and the initial game model along the optimal cooperative trajectory $x^{*}(t)$ is demonstrated in Figure 4.

Figure 4 shows that if we consider the problem in a more realistic case (continuous updating), a player with continuous updating can get less allocation from the coalition than they get from the initial game model. This is based on the fact that, at an early stage, the pollution emitted into the atmosphere is more than in the initial game model, and in the latter stage the pollution with continuous updating is less than in the initial game model. Thus, starting with $t=0$, players get more in the initial game model, but in the same period, they all get 0 in the end. This shows that as pollution intensifies the benefits countries receive from its attendant production gradually decrease.

5. Conclusions

In this paper, we presented the detailed consideration of a cooperative differential game model with continuous updating based on Pontryagin maximum principle, where the decision-maker updates his/her behavior based on the new information available which arises from a shifting time horizon. The characteristic function with continuous updating obtained by using the Pontryagin maximum principle for the cooperative case is constructed. The results show that the $\delta$-characteristic function computed for the game is superadditive and does not have any other restrictions on the model’s parameters. The concept of the Shapley value as a cooperative solution with continuous updating is demonstrated in an analytic form for pollution control problems. Ultimately, considering the example of n-player pollution control, optimal strategies, the corresponding trajectory, the characteristic function, and the Shapley value with continuous updating are conceived for the proposed application and graphically compared for their effectiveness. We showed simulation results that show the applicability of the approach.

The practical significance of the work is determined by the fact that the real life conflict controlled processes evolve continuously in time and the players usually are not or cannot use full information about it. Therefore, it is important to introduce the type of differential games with information updating to the field of game theory. Another important practical contribution of the continuous updating approach is the creation of a class of inverse optimal control problems with continuous updating [17]. Problems that can be used to analyze a profile of the human in the human-machine type of engineering systems. The results are illustrated on the model of a driver assistance system and are applied to the real driving data from the simulator located in the Institute of Control Systems, Karlsruhe Institute of Technology. Our method can provide more in-depth modeling of human engineering systems.