Appendix
Proof of Theorem 1. Introduce a function \({\Phi }_{0}:{{\mathbb{R}}}^{n}\to {\mathbb{R}}\) such that \({\Phi }_{0}\left(x\right)={\Phi }_{1}\left(x\right)\). Augment the state vector of the system by adding a new variable \({y}_{k}={\rm{min}}_{i = \overline{0,k}}{\Phi }_{i}\left({x}_{i}\right)\). The augmented control system has the form
$$\left\{\begin{array}{l}{x}_{k+1}={f}_{k}\left({x}_{k},{\widetilde{u}}_{k},{\xi }_{k}\right)\\ {y}_{k+1}={\rm{min}}\left\{{y}_{k},{\Phi }_{k}\left({x}_{k}\right)\right\}\\ {x}_{0}=X\\ {y}_{0}={\Phi }_{0}\left(X\right),\end{array}\right.\qquad k=\overline{0,N},$$
where \({\widetilde{u}}_{k}={\widetilde{\gamma }}_{k}\left({x}_{k},{y}_{k}\right)\). Consider the right-hand side \(\widetilde{f}:{{\mathbb{R}}}^{n+1}\times {{\mathbb{R}}}^{m}\times {{\mathbb{R}}}^{s}\to {{\mathbb{R}}}^{n+1}\) of the augmented system:
$${\widetilde{f}}_{k}\left(x,y,u,\xi \right)={\left({f}_{k}\left(x,u,\xi \right),{\rm{min}}\left\{y,{\Phi }_{k}\left(x\right)\right\}\right)}^{\,\,\,{\mathtt{T}}}.$$
Then the equivalent optimal control problem can be written as
$${\bf{P}}\left({\rm{min}}\left\{{y}_{N+1},{\Phi }_{N+1}\left({x}_{N+1}\right)\right\}\leqslant \varphi \right)\to \mathop{\rm{sup}}\limits_{u\left(\cdot \right)\in \widetilde{{\mathcal{U}}}},\quad k=\overline{0,N},$$
where \(\widetilde{{\mathcal{U}}}={\widetilde{{\mathcal{U}}}}_{0}\times \ldots \times {\widetilde{{\mathcal{U}}}}_{N}\) and
$${\widetilde{{\mathcal{U}}}}_{k}=\left\{\widetilde{\gamma }:{{\mathbb{R}}}^{n+1}\to {{\mathbb{R}}}^{m}| \widetilde{\gamma }\,\text{is Borel measurable}\,,\,\forall x\in {{\mathbb{R}}}^{n+1}:\widetilde{\gamma }\left(x\right)\in {U}_{k}\right\}.$$
Here the equivalence is understood in the sense of the same criteria
$$P\left(u\left(\cdot \right)\right)={\bf{P}}\left(\mathop{\rm{min}}\limits_{k = \overline{0,N}}{\Phi }_{k+1}\left(x\right)\leqslant \varphi \right)={\bf{P}}\left(\mathop{\rm{min}}\limits\left\{{y}_{N+1},{\Phi }_{N+1}\left({x}_{N+1}\right)\right\}\leqslant \varphi \right).$$
The Bellman equation for the equivalent problem [12] has the form
$${\widetilde{\gamma }}_{k}^{* }\left(x,y\right)={\rm{arg}}\,\mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left[{\widetilde{{\rm{B}}}}_{k+1}\left({\widetilde{f}}_{k}\left(x,y,u,{\xi }_{k}\right)\right)\right],$$
(A.1)
$${\widetilde{{\rm{B}}}}_{k}\left(x,y\right)=\mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left[{\widetilde{{\rm{B}}}}_{k+1}\left({\widetilde{f}}_{k}\left(x,y,{u}_{k},{\xi }_{k}\right)\right)\right],$$
(A.2)
$${\widetilde{{\rm{B}}}}_{N+1}\left(x,y\right)={{\bf{I}}}_{\left\{{\rm{min}}\left\{y,{\Phi }_{N+1}\left(x\right)\right\}\leqslant \varphi \right\}}\left(x,y\right),\quad k=\overline{0,N}.$$
(A.3)
The following result was established in [12]. Assume that:
1) The functions \({\widetilde{f}}_{k}\) are continuous for all \(k=\overline{0,N}\).
2) The function \({\rm{min}}\left\{y,{\Phi }_{N+1}\left(x\right)\right\}\) is continuous and bounded below.
3) The random vectors X, ξ0, …, ξN are independent.
4) The sets U0, …, UN are compact.
Then the optimal control exists in the class of measurable functions and is determined by solving problems (A.1)–(A.3).
At step k = N the Bellman equation takes the form
$$\begin{array}{rcl}{\widetilde{{\rm{B}}}}_{N}\left(x,y\right)&=&\mathop{\rm{max}}\limits_{u\in {U}_{N}}{{\bf{M}}}_{{\xi }_{N}}\left[{{\bf{I}}}_{\left\{{\rm{min}}\left\{y,{\Phi }_{N}\left(x\right),{\Phi }_{N+1}\left(f\left(x,u,{\xi }_{k}\right)\right)\right\}\leqslant \varphi \right\}}\left(x,y\right)\right]\\ &=&\mathop{\rm{max}}\limits_{u\in {U}_{N}}{{\bf{M}}}_{{\xi }_{N}}\left[{{\bf{I}}}_{\left\{{\rm{min}}\left\{y,{\Phi }_{N}\left(x\right)\right\}\leqslant \varphi \right\}}\left(x,y\right)+\left(1-{{\bf{I}}}_{\left\{{\rm{min}}\left\{y,{\Phi }_{N}\left(x\right)\right\}\leqslant \varphi \right\}}\left(x,y\right)\right)\right.\\ &&\hspace{1em}\left.\times {{\bf{I}}}_{\left\{{\Phi }_{N+1}\left(f\left(x,u,{\xi }_{k}\right)\right)\leqslant \varphi \right\}}\left(x\right)\right]\\ &=&\mathop{\rm{max}}\limits_{u\in {U}_{N}}{{\bf{M}}}_{{\xi }_{N}}\left[{{\bf{I}}}_{\left\{{\rm{min}}\left\{y,{\Phi }_{N}\left(x\right)\right\}\leqslant \varphi \right\}}\left(x,y\right)+\left(1-{{\bf{I}}}_{\left\{{\rm{min}}\left\{y,{\Phi }_{N}\left(x\right)\right\}\leqslant \varphi \right\}}\left(x,y\right)\right)\right.\\ &&\hspace{1em}\left.\times {{\rm{B}}}_{N+1}\left({f}_{N}\left(x,u,{\xi }_{N}\right)\right)\right].\end{array}$$
(A.4)
Hence, for any \(k=\overline{0,N}\), Eq. (A.2) can be written as
$$\begin{array}{l}{{\rm{B}}_k}\left( {x,y} \right) = \mathop {{\rm{max}}}\limits_{u \in {U_k}} {{\bf{M}}_{{\xi _k}}}[{{\bf{I}}_{{\rm{\{ min}}\left\{ {{\rm{y}},{\Phi _{\rm{k}}}\left( {\rm{x}} \right)} \right\}\varphi \} }}\left( {{\rm{x}},{\rm{y}}} \right)\\\quad\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\left. {{\rm{ + }}\left( {{\rm{1 - }}{{\bf{I}}_{\left\{ {{\rm{min}}\left\{ {{\rm{y}},{\Phi _{\rm{k}}}\left( {\rm{x}} \right)} \right\}\varphi } \right\}}}\left( {{\rm{x}},{\rm{y}}} \right)} \right){{\rm{B}}_{{\rm{k + 1}}}}\left( {{{\rm{f}}_{\rm{k}}}\left( {{\rm{x}},{\rm{u}},{\xi _{\rm{k}}}} \right)} \right)} \right].\end{array}$$
(A.5)
Therefore, the Bellman function in the equivalent problem admits of the representation
$${\widetilde{{\rm{B}}}}_{k}(x,y)=\left\{\begin{array}{ll}1,&y\leqslant \varphi \\ \mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left\{{{\bf{I}}}_{{{\mathcal{F}}}_{k}}(x)+(1-{{\bf{I}}}_{{{\mathcal{F}}}_{k}}(x)){{\rm{B}}}_{k+1}({f}_{k}(x,u,{\xi }_{k}))\right\},&y>\varphi .\end{array}\right.$$
(A.6)
As a result, the optimal control is given by
$${\widetilde{\gamma }}_{k}^{* }(x,y)=\left\{\begin{array}{ll}\,\text{any element from}\,\,{U}_{k},&y\leqslant \varphi \\ {\rm{arg}}\,\mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left\{{{\bf{I}}}_{{{\mathcal{F}}}_{k}}(x)+(1-{{\bf{I}}}_{{{\mathcal{F}}}_{k}}(x)){{\rm{B}}}_{k+1}({f}_{k}(x,u,{\xi }_{k}))\right\},&y>\varphi .\end{array}\right.$$
(A.7)
Conditions 1–5 of Theorem 1 follow from the existence conditions of an optimal control in the problem with the terminal criterion. Note that items 2 and 3 are immediate from the continuity and boundedness (from below) of the function \({\rm{min}}\left\{y,{\Phi }_{N+1}\left(x\right)\right\}\).
The proof of Theorem 1 is complete.
Proof of Theorem 2. 1. Consider the Bellman equation at some step k. For the isobell of level 1,
$$\begin{array}{ccc}{{\mathcal{I}}}_{k}&=&\left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left[{{\bf{I}}}_{{{\mathcal{F}}}_{k}}\left(x\right)+\left(1-{{\bf{I}}}_{{{\mathcal{F}}}_{k}}\left(x\right)\right){{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)\right]=1\right\}\\ &=&{{\mathcal{F}}}_{k}\cup \left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)\right]=1\right\}.\end{array}$$
Using the total mathematical expectation formula, write
$$\begin{array}{cc}{{\mathcal{I}}}_{k}={{\mathcal{F}}}_{k}\cup \left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}\left\{{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right){{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right]\right.\right.\\ \left.\left.+\left(1-{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)\right){{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\notin {{\mathcal{I}}}_{k+1}\right]\right\}=1\right\}.\end{array}$$
From the equalities
$$\begin{array}{l}{{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right]=1,\\ {{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\notin {{\mathcal{I}}}_{k+1}\right]<1,\end{array}$$
it follows that
$$\begin{array}{ccc}{{\mathcal{I}}}_{k}={{\mathcal{F}}}_{k}\cup \left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}\left\{{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)+\left(1-{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)\right)\right.\right.\\ \left.\left.\times {{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\notin {{\mathcal{I}}}_{k+1}\right]\right\}=1\right\}\\ ={{\mathcal{F}}}_{k}\cup \left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)=1\right\}.\end{array}$$
Item 1 of this theorem is established.
2. By analogy with item 1, the isobell of level 0 of the Bellman function can be written as
$$\begin{array}{ccc}{{\mathcal{O}}}_{k}&=&\left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left[{{\bf{I}}}_{{{\mathcal{F}}}_{k}}\left(x\right)+\left(1-{{\bf{I}}}_{{{\mathcal{F}}}_{k}}\left(x\right)\right){{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)\right]=0\right\}\\ &=&{\overline{{\mathcal{F}}}}_{k}\cap \left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}{{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)\right]=1\right\}.\end{array}$$
Using the total mathematical expectation formula, transform the right-hand side of this expression:
$$\begin{array}{rcl}{{\mathcal{O}}}_{k}&=&{\overline{{\mathcal{F}}}}_{k}\cap \left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}\left\{{{\bf{P}}}_{{\xi }_{k}}({f}_{k}(x,u,{\xi }_{k})\in {{\mathcal{I}}}_{k+1}){{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}({f}_{k}(x,u,{\xi }_{k}))| {f}_{k}(x,u,{\xi }_{k})\in {{\mathcal{I}}}_{k+1}\right]\right.\right.\\ &&+{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right){{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right]\\ &&\left.\left.+{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{O}}}_{k+1}\right){{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{O}}}_{k+1}\right]\right\}=0\right\}.\end{array}$$
From the equalities
$$\begin{array}{ccc}{{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right]=1,\\ {{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right]\in \left(0,1\right),\\ {{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{O}}}_{k+1}\right]=0\end{array}$$
it follows that
$$\begin{array}{l}{{\mathcal{O}}}_{k}={\overline{{\mathcal{F}}}}_{k}\cap \left\{x\in {{\mathbb{R}}}^{n}| \mathop{\rm{max}}\limits_{u\in {U}_{k}}\left\{{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)\right.\right.\\ \left.\left.\quad\quad\quad\quad+{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right){{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right]\right\}=0\right\}\\ \quad\quad={\overline{{\mathcal{F}}}}_{k}\cap \left\{x\in {{\mathbb{R}}}^{n}| \forall u\in {U}_{k}:{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)=0,\,{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right)=0\right\}\\ \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad={\overline{{\mathcal{F}}}}_{k}\cap \left\{x\in {{\mathbb{R}}}^{n}| \forall u\in {U}_{k}:{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{O}}}_{k+1}\right)=1\right\}.\end{array}$$
Item 2 of this theorem is established.
3. Item 3 of this theorem follows from its item 1.
4. Item 4 of this theorem holds, since \({{\rm{B}}}_{k}\left(x\right)=0\) for \(x\in {{\mathcal{O}}}_{k}\).
5. Item 5 of this theorem follows its item 1.
6. For \(x\in {{\mathcal{B}}}_{k}\), the right-hand side of the dynamic programming equation can be represented as
$$\begin{array}{ccc}{{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)\right]={{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)\\ \times \left(1-{{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right]\right)\\ +\left(1-{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{O}}}_{k+1}\right)\right){{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)| {f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{B}}}_{k+1}\right].\end{array}$$
Due to the two-sided inequality for a convex combination, this yields
$$\begin{array}{cc}{\rm{min}}\left\{{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right),\left(1-{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{O}}}_{k+1}\right)\right)\right\}{{\bf{M}}}_{{\xi }_{k}}\left[{{\rm{B}}}_{k+1}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)\right]\\ \leqslant {\rm{max}}\left\{{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right),\left(1-{{\bf{P}}}_{{\xi }_{k}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{O}}}_{k+1}\right)\right)\right\}.\end{array}$$
The relations \(1-{{\bf{P}}}_{{\xi }_{k}}({f}_{k}(x,u,{\xi }_{k})\in {{\mathcal{O}}}_{k+1})={{\bf{P}}}_{{\xi }_{k}}({f}_{k}(x,u,{\xi }_{k})\in {{\mathcal{I}}}_{k+1})+{{\bf{P}}}_{{\xi }_{k}}({f}_{k}(x,u,{\xi }_{k})\in {{\mathcal{B}}}_{k+1})\) and \({{\mathcal{F}}}_{k}\subseteq {{\mathcal{I}}}_{k}\) finally establish item 6 of this theorem.
7. Item 7 of this theorem is immediate from its item 6 by taking the supremum in all sides of inequality (8).
Proof of Theorem 3. 1. Introduce a system of hypotheses forming a complete group of incompatible events:
$$\left\{\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right\},\quad \left\{\mathop{\bigcap}\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right\}.$$
Then the total probability formula gives the chain of equalities
$$\begin{array}{ccc}P\left(\underline{u}\left(\cdot \right)\right)={\bf{P}}\left(\mathop{\bigcup }\limits_{k = 0}^{N}\left\{{\underline{x}}_{k+1}\in {{\mathcal{F}}}_{k+1}\right\}\right)\\ ={\bf{P}}\left(\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 0}^{N}\left\{{\underline{x}}_{k+1}\in {{\mathcal{F}}}_{k+1}\right\}\right|\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)\\ +\,\,{\bf{P}}\left(\mathop{\bigcap}\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 0}^{N}\left\{{\underline{x}}_{k+1}\in {{\mathcal{F}}}_{k+1}\right\}\right|\mathop{\bigcap}\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right).\end{array}$$
(A.8)
Analyze the second factor in the first term of the right-hand side of this expression. From the chain of equalities
$${\bf{P}}\left(\left.{\underline{x}}_{N+1}\in {{\mathcal{F}}}_{N+1}\right|{\underline{x}}_{N}\in {{\mathcal{I}}}_{N}\right)={\bf{P}}\left(\left.{\underline{x}}_{N+1}\in {{\mathcal{I}}}_{N+1}\right|{\underline{x}}_{N}\in {{\mathcal{I}}}_{N}\right)=1$$
it follows that
$${\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 0}^{N}\left\{{\underline{x}}_{k+1}\in {{\mathcal{F}}}_{k+1}\right\}\right|\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)=1.$$
In view of this equality, the expression (A.8) takes the form
$$\begin{array}{cc}P\left(\underline{u}\left(\cdot \right)\right)={\bf{P}}\left(\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)\\ +\,\,{\bf{P}}\left(\mathop{\bigcap }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 0}^{N}\left\{{\underline{x}}_{k+1}\in {{\mathcal{F}}}_{k+1}\right\}\right|\mathop{\bigcap }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right).\end{array}$$
(A.9)
Now introduce another system of hypotheses forming a complete group of incompatible events:
$$\left\{\mathop{\bigcup }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right\},\quad \left\{\mathop{\bigcap }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right\}.$$
Apply the total probability formula for the first term in the right-hand side of (A.9):
$$\begin{array}{ccc}{\bf{P}}\left(\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)\\ ={\bf{P}}\left(\mathop{\bigcup }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right|\mathop{\bigcup }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)\\ +\,\,{\bf{P}}\left(\mathop{\bigcap }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right|\mathop{\bigcap }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right).\end{array}$$
(A.10)
Similar to (A.9), transform the right-hand side of (A.10) to obtain
$$\begin{array}{cc}{\bf{P}}\left(\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)={\bf{P}}\left(\mathop{\bigcup }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)\\ +\,\,{\bf{P}}\left(\mathop{\bigcap }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right|\mathop{\bigcap }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right).\end{array}$$
(A.11)
Substituting (A.10) into (A.9) yields
$$\begin{array}{ccc}P\left(\underline{u}\left(\cdot \right)\right)={\bf{P}}\left(\mathop{\bigcup }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right)\\ +\,\,{\bf{P}}\left(\mathop{\bigcap }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right|\mathop{\bigcap }\limits_{k = 1}^{N-1}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right)\\ +\,\,{\bf{P}}\left(\mathop{\bigcap }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 0}^{N}\left\{{\underline{x}}_{k+1}\in {{\mathcal{F}}}_{k+1}\right\}\right|\mathop{\bigcap }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right).\end{array}$$
(A.12)
Perform the analogous transformations for the first term in (A.12), introducing the systems of hypotheses
$$\left\{\mathop{\bigcup }\limits_{k = 1}^{l}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right\},\quad \left\{\mathop{\bigcap }\limits_{k = 1}^{l}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right\},\quad l=\overline{1,\ldots ,N-2},$$
to obtain the following expression for the value of the probabilistic criterion under the control \(\underline{u}\left(\cdot \right):\)
$$\begin{array}{ccc}P\left(\underline{u}\left(\cdot \right)\right)={\bf{P}}\left({\underline{x}}_{1}\in {{\mathcal{I}}}_{1}\right)\\ +\mathop{\sum }\limits_{l = 1}^{N-1}{\bf{P}}\left(\mathop{\bigcap }\limits_{k = 1}^{l}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 1}^{l+1}\left\{{\underline{x}}_{k}\in {{\mathcal{I}}}_{k}\right\}\right|\mathop{\bigcap }\limits_{k = 1}^{l}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right)\\ +\,\,{\bf{P}}\left(\mathop{\bigcap }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right){\bf{P}}\left(\left.\mathop{\bigcup }\limits_{k = 0}^{N}\left\{{\underline{x}}_{k+1}\in {{\mathcal{F}}}_{k+1}\right\}\right|\mathop{\bigcap }\limits_{k = 1}^{N}\left\{{\underline{x}}_{k}\notin {{\mathcal{I}}}_{k}\right\}\right).\end{array}$$
(A.13)
Note that the first term in (A.13) satisfies the chain of equalities
$${\bf{P}}\left({\underline{x}}_{1}\in {{\mathcal{I}}}_{1}\right)={\bf{P}}\left({f}_{0}\left(X,{\underline{u}}_{0},{\xi }_{0}\right)\in {{\mathcal{I}}}_{1}\right)={\underline{{\rm{B}}}}_{0}\left(X\right)=\underline{F}\left(\varphi ,N,X\right),$$
which implies the expression (14).
Item 1 of Theorem 3 is established.
2. For establishing item 2 of Theorem 3, it suffices to observe that for \({x}_{k}\in {{\mathcal{I}}}_{k}\), \({u}_{k}^{* }={\underline{u}}_{k}\) for all \(k=\overline{0,N}\). Following the same procedure as above, this equality can be used for deriving the expression (14) for the optimal-value function of the probabilistic criterion on the trajectories \({\left\{{x}_{k}^{* }\right\}}_{k = 1}^{N+1}\) of the closed loop system with the optimal control \({u}^{* }\left(\cdot \right)\).
Item 2 of Theorem 3 is established.
3. Item 3 of Theorem 3 directly follows from item 7 of Theorem 2 and item 1 of Theorem 3.
The proof of Theorem 3 is complete.
Proof of Lemma. Consider the dynamic programming relations (3)–(5) and (18)–(20) for problems (2) and (17), respectively. In (19), the equality
$${{\rm{B}}}_{k}^{\varphi }\left(x\right)=\mathop{\rm{max}}\limits_{u\in {U}_{k}}{\bf{M}}\left[{{\bf{I}}}_{{{\mathcal{I}}}_{k}^{\varphi }}\left(x\right)+\left(1-{{\bf{I}}}_{{{\mathcal{I}}}_{k}^{\varphi }}\left(x\right)\right){{\rm{B}}}_{k+1}^{\varphi }\left({f}_{k}\left(x,u,{\xi }_{k}\right)\right)\right],\quad k=\overline{0,N},$$
where \({{\mathcal{I}}}_{k}^{\varphi }\) is the isobell of level 1 of the Bellman function \({{\rm{B}}}_{k}^{\varphi }\left(x\right)\), holds for all \(x\in {{\mathbb{R}}}^{n}\). Hence, the equivalence conditions are true if the isobells of level 1 of the Bellman functions in problems (2) and (17) are the same, \({{\mathcal{I}}}_{k}={{\mathcal{I}}}_{k}^{\varphi }\). Due to the recurrent formula for \({{\mathcal{I}}}_{k}\) (see item 1 of Theorem 2), this is the case if and only if
$${{\mathcal{F}}}_{k}\subseteq \left\{x\in {{\mathbb{R}}}^{n}:\exists u\in {U}_{k}:{\bf{P}}\left({f}_{k}\left(x,u,{\xi }_{k}\right)\in {{\mathcal{I}}}_{k+1}\right)=1\right\},\quad k=\overline{0,N}.$$
(A.14)
The proof of Lemma is complete.
Proof of Proposition 1. In accordance with item 1 of Theorem 2, write the equations for the isobells of levels 1 and 0 of the Bellman function at step k = N:
$$\begin{array}{l}{{\mathcal{I}}}_{N}={\mathcal{F}}\cup \left\{x\in {\mathbb{R}}:\exists u\in U:{\bf{P}}\left(x\left(1+{u}^{1}b+\mathop{\sum }\limits_{i=2}^{m}{u}^{i}{\xi }_{N}^{i}\right)\geqslant -\varphi \right)=1\right\},\\ {{\mathcal{O}}}_{N}=\overline{{\mathcal{F}}}\cap \left\{x\in {\mathbb{R}}:\forall u\in U:{\bf{P}}\left(x\left(1+{u}^{1}b+\mathop{\sum }\limits_{i=2}^{m}{u}^{i}{\xi }_{N}^{i}\right)<-\varphi \right)=1\right\}.\end{array}$$
The solutions of these equations were obtained in [13]. Using those results and the notations for the boundaries of the sets \({{\mathcal{I}}}_{k}\) and \({{\mathcal{O}}}_{k}\) (see Section 6), write:
$$\begin{array}{l}{{\mathcal{I}}}_{N}=\left[\varphi ,+\infty \right)\cup \left[\varphi {\left(1+{\rm{max}}\left\{b,\mathop{\rm{max}}\limits_{j = \overline{2,m}}{\underline{b}}^{j}\right\}\right)}^{-1},+\infty \right)=\left[{\varphi }_{N}^{{\mathcal{I}}},+\infty \right),\\ {{\mathcal{O}}}_{N}=\left(-\infty ,\varphi \right]\cap \left(-\infty ,\varphi {\left(1+{\rm{max}}\left\{b,\mathop{\rm{max}}\limits_{j = \overline{2,m}}{\overline{b}}^{j}\right\}\right)}^{-1}\right]=\left(-\infty ,{\varphi }_{N}^{{\mathcal{O}}}\right].\end{array}$$
This implies \({{\mathcal{I}}}_{N}=\Delta {{\mathcal{I}}}_{N}\). Due to item 1 of Theorem 2, at step k = N − 1 the equations for the isobells take the form
$$\begin{array}{l}{{\mathcal{I}}}_{N-1}={\mathcal{F}}\cup \left\{x\in {\mathbb{R}}:\ \exists u\in U:\ {\bf{P}}\left(x\left(1+{u}^{1}b+\mathop{\sum }\limits_{i = 2}^{m}{u}^{i}{\xi }_{N-1}^{i}\right)\geqslant {\varphi }_{N}^{{\mathcal{I}}}\right)=1\right\},\\ {{\mathcal{O}}}_{N-1}=\overline{{\mathcal{F}}}\cap \left\{x\in {\mathbb{R}}:\forall u\in U:{\bf{P}}\left(x\left(1+{u}^{1}b+\mathop{\sum }\limits_{i = 2}^{m}{u}^{i}\ {\xi }_{N-1}^{i}\right)<{\varphi }_{N}^{{\mathcal{O}}}\right)=1\right\}.\end{array}$$
By analogy with step k = N, it follows that
$${{\mathcal{I}}}_{N-1}=\Delta {{\mathcal{I}}}_{N-1}=\left[{\varphi }_{N-1}^{{\mathcal{I}}},+\infty \right),\quad {{\mathcal{O}}}_{N-1}=\left(-\infty ,{\varphi }_{N-1}^{{\mathcal{O}}}\right].$$
Mathematical induction on k finally leads to the conclusion that, for all \(k=\overline{0,N}\),
$${{\mathcal{I}}}_{k}=\Delta {{\mathcal{I}}}_{k}=\left[{\varphi }_{k}^{{\mathcal{I}}},+\infty \right),\quad {{\mathcal{O}}}_{k}=\left(-\infty ,{\varphi }_{k}^{{\mathcal{O}}}\right].$$
The proof of Proposition 1 is complete.
Proof of Proposition 2. Consider the control (11), which takes the form (25); see Proposition 1. From item 3 of Theorem 3 it follows that
$$P\left(\underline{u}\left(\cdot \right)\right)={\bf{P}}\left(\mathop{\max }\limits_{k\in \left\{0,\ldots ,N\right\}}{\underline{x}}_{k+1}\geqslant -\varphi \right)\geqslant \underline{F}\left(\varphi ,N,X\right),$$
(A.15)
where the function \(\underline{F}\) is given by
$$\underline{F}\left(\varphi ,N,X\right)=\mathop{\rm{max}}\limits_{u\in U}{\bf{P}}\left(X\left(1+b{u}^{1}+\mathop{\sum }\limits_{j = 2}^{m}{u}^{j}{\xi }_{0}^{j-1}\right)\geqslant -\frac{\varphi }{{\left(1+{\rm{max}}\left\{b,\mathop{\rm{max}}\limits_{j = \overline{1,m-1}}{\underline{\varepsilon }}_{j}\right\}\right)}^{N}}\right)$$
(A.16)
due to
$${\varphi }_{1}^{{\mathcal{I}}}=-\varphi {\left(1+{\rm{max}}\left\{b,\mathop{\rm{max}}\limits_{j = \overline{1,m-1}}{\underline{\varepsilon }}_{j}\right\}\right)}^{-N}.$$
Clearly, the value \(\underline{N}\in {\mathbb{N}}\) can be determined as a root of the equation \(\underline{F}\left(\varphi ,N,X\right)=1\). But there exist infinitely many such roots, and the estimate \(\underline{N}\) will be therefore found in the form
$$\underline{N}={\rm{min}}\left\{N\in {\mathbb{N}}:\underline{F}\left(\varphi ,N,X\right)=1\right\}.$$
(A.17)
The expressions (A.15) and (A.17) imply
$${\bf{P}}\left(\mathop{\rm{max}}\limits_{k\in \left\{0,\ldots ,\underline{N}\right\}}{\underline{x}}_{k+1}\geqslant -\varphi \right)=1.$$
From the definition of the isobell of level 1 of the Bellman function and the definition of the function \(\underline{F}\) it follows that the equation \(\underline{F}\left(\varphi ,N,X\right)=1\) is equivalent to the inclusion \(X\in {{\mathcal{I}}}_{0}\), which is in turn equivalent to
$$X\geqslant -\frac{\varphi }{{\left(1+{\rm{max}}\left\{b,\mathop{\rm{max}}\limits_{j = \overline{1,m-1}}{\underline{\varepsilon }}_{j}\right\}\right)}^{N+1}}.$$
Taking the logarithm gives the inequality
$$N\geqslant \frac{\mathrm{ln}\,\left(-\varphi \right)-\mathrm{ln}\,\left(X\right)}{\mathrm{ln}\,\left(1+{\rm{max}}\left\{b,\mathop{\rm{max}}\limits_{j = \overline{1,m-1}}{\underline{\varepsilon }}_{j}\right\}\right)}-1.$$
In view of (A.17), this finally leads to (26).
The proof of Proposition 2 is complete.