Continue, quit, restart probability model

Abstract

We discuss a new applied probability model: there is a system whose evolution is described by a Markov chain (MC) with known transition matrix on a discrete state space and at each moment of a discrete time a decision maker can apply one of three possible actions: continue, quit, and restart MC in one of a finite number of fixed “restarting” points. Such a model is a generalization of a model due to Katehakis and Veinott (Math. Oper. Res. 12:262, 1987), where a restart to a unique point was allowed without any fee and quit action was absent. Both models are related to Gittins index and to another index defined in a Whittle family of stopping retirement problems. We propose a transparent recursive finite algorithm to solve our model by performing O(n ³) operations.

Appendix

1.1 5.1 Three abstract optimization problems

The common part of all three problems described above in Sect. 3 is a maximization over the set of all partitions of the state set X into three sets: continuation, restart and quit regions. This is a special case of a very general situation of optimization on an abstract index set.

Let us consider the following three abstract optimization problems 1, 2 and 3. Suppose there is an abstract index set U, and let A={a _u } and B={b _u } be two sets of real numbers indexed by the elements of U. Suppose that an assumption U holds,

$$-\infty<a_{u}\leq a<\infty,\qquad 0<b\leq b_{u}\leq1. $$

(U)

In all three problems a DM knows sets U,A and B.

Problem 1: restart problem.:

Find solution(s) of the equation

$$h=\sup_{u\in U}\bigl[a_{u}+(1-b_{u})h\bigr]\equiv H(h). $$

(41)

Equation (41) is a Bellman (optimality) equation for the “value of the game”, i.e. the supremum over all possible strategies in the optimization problem with two equivalent interpretations. In both cases set U represents a set of available actions. A DM can select one of them, say u, and then she obtains a reward a _u and, according to the first interpretation with probability b _u , the game is terminated, and with complimentary probability 1−b _u she is again in an initial situation, i.e. she can select any action. Her goal is to maximize the total (undiscounted) reward. According to the second interpretation, the game is continued sequentially without possibility of random termination, but the value 1−b _u is now not a probability, but a discount factor applied to the future rewards after an action u was used at the first step.

Problem 2: ratio (cycle) problem.:

Find

$$\alpha=\sup_{u\in U}\frac{a_{u}}{b_{u}}. $$

(42)

The interpretation of this problem is straightforward: a DM can select action u only once and her goal is to maximize the ratio in (42), the one step reward per “chance of termination”. Since the game is terminated after the first trial anyway, 1/b _u has an interpretation of a “multiplicator” applied to a “direct” reward a _u .

Let H(k) be a function defined in the right side of (41).

Problem 3: a parametric family of retirement problems.:

Find w, defined as follows: given parameter k,−∞<k<∞, let

$$v(k)=k\vee H(k),\qquad w=\inf\bigl\{k:v(k)=k\bigr\}. $$

(43)

In this problem, given number k, a DM has the following one step choice: to obtain k immediately, or to select an action u once and then to obtain a reward a _u and after that additionally with probability 1−b _u to obtain k, and with complimentary probability to obtain zero.

Using the fact that functions H(k) and v(k),−∞<k<∞, are nondecreasing, continuous, and convex (concave up), the following theorem was proved in Sonin (2011).

Theorem 5

(Abstract optimization equality)

• (a) Solution h of (41) is finite and unique;

• (b) h=α=w;

• (c) the optimal index, or an optimizing sequence for any of the three problems is the optimal index (an optimizing sequence) for the other two problems.

See the brief discussion of one more problem initially analyzed in one page seminal paper Mitten (1960), and its relation to the GI and GGI in Sonin (2008).

In the next section we shall formally show how the three problems described in Sect. 3 can be presented as abstract problems.

1.2 5.2 Proofs of some statements

Proof of Proposition 3

For any set S⊂X by definition of τ=τ _S and definition of function g(x|k) we have

(44)

Using the definitions of R ^π(x) and Q ^π(x) in (26) and (25), the definition of function g(x|k) and the definition of a partition π ₀, we can see that for all partitions π with the same set S we have $C^{\pi}(x)=C^{\pi_{0}}(x)$, $R_{q}^{\pi }(x)+R_{r}^{\pi}(x)+Q_{r}^{\pi}(x)k\leq R_{q}^{\pi_{0}}(x)+R_{r}^{\pi _{0}}(x)+Q_{r}^{\pi_{0}}(x)k$. Hence R ^π(x)+(1−Q ^π(x))k≤ $R^{\pi_{0}}(x)+(1-Q^{\pi_{0}}(x))k$. Thus sup _π>0[…]=sup _τ>0[…], where [..] is the expression in square brackets in (36). □

Proof of Theorem 3, using Theorem 4

Given x∈X let us introduce the set of indices U={u}={π>0}∪{0}, where π>0 is any 3-partition such that x∈C and 0 is an index such that a ₀=q(x)−r(x),b ₀=1. For any u≠0, i.e. for a partition π=(C,S _q ,S _r ), S=S _q ∪S _r , and τ=τ _S , we define $R^{\pi}(x)=E_{x}\sum_{n=0}^{\tau-1}c(Z_{n})$, the total expected reward until moment τ, and Q(x)=P _x (Z _τ ∈S _q ), the probability of termination on [0,τ). The rewards a _u and probabilities b _u we define as a _u =R ^π(x)−r(x), b _u =Q ^π(x). Then function H(k) coincides with sup _τ>0 E _x g(Z _τ ), where g(x)=k. Respectively v(x|k)=k∨H(k)=sup _τ≥0 E _x g(Z _τ ), i.e. v(x|k) is the value function in an OS for MC in model M(k). Then according to formulas (29), (30) and (33) and (34) three Problems 1, 2, 3 from Sect. 4 are represented as three abstract problems in this section and we can apply Theorem 4. □

Proof of formula (15)

Note that if D=∅ then W _∅=P _∅=P and if D={y} then p _D╲y (⋅,y)=p(⋅,y) and formula (14) with w′=w _D and w=p for z=y implies that

$$w_{D}(\cdot,y)=p(\cdot,y)+p(\cdot,y)\frac{p(y,y)}{1-p(y,y)}=\frac {p(\cdot,y)}{1-p(y,y)},$$

(45)

so formula (15) holds. It remains to prove that if it holds for a set D⊂X, then it will hold for a set D∪z, z∉D. For D⊂X let us define matrix S _D as follows: s _D (⋅,y)=w _D (⋅,y)=p _D (⋅,y) for y∉D and s _D (⋅,y)=p _D╲y (⋅,y) for y∈D. To simplify notation let us denote matrices W _D =W ₁,W _D∪z =W ₂,S _D =S ₁,S _D∪z =S ₂. If formula (15) holds for a set D⊂X, then two equivalent statements are true

$$w_{1}(\cdot,y)=\frac{s_{1}(\cdot,y).}{1-s_{1}(y,y)},\qquad s_{1}(\cdot,y)=\frac {w_{1}(\cdot,y).}{1+w_{1}(y,y)},\quad y\in D. $$

(46)

The first formula is our assumption and the second formula follows from the first one if we apply the first formula for x=y to obtain the equality 1−s ₁(y,y)=1/(1+w ₁(y,y)). Our goal is to prove

$$w_{2}(\cdot,y)=\frac{s_{2}(\cdot,y).}{1-s_{2}(y,y)},y\in D\cup z. $$

(47)

For y=z formula (47) can be checked directly: formula (14) for y=z implies w ₂(⋅,z)=w ₁(⋅,z)/(1−w ₁(z,z)) and for z∉D by definition of S ₂ we have s ₂(⋅,z)=s ₁(⋅,z), and by definition of S ₁ we have s ₁(⋅,z)=w ₁(⋅,z). For y∈D by definition of W ₂, i.e. by formula (14), using formula (46) and the equality w ₁(⋅,z)=s ₁(⋅,z) (since z∉D) we have

(48)

By definition of S ₂=S _D∪z , using formula (14) with w′=p _(D∪z)╲y and w=p _D╲y for y∈D we have

$$s_{2}(\cdot,y)=s_{1}(\cdot,y)+p_{D\diagdown y}(\cdot ,z)s_{1}(z,y)A, $$

(49)

where A=1/(1−p _D╲y (z,z)). Using formula (14) with w′=p _D and w=p _D╲y , we have

$$p_{D}(\cdot,z)=p_{D\diagdown y}(\cdot,z)+p_{D\diagdown y}(\cdot,y)\frac {p_{D\diagdown y}(y,z)}{1-p_{D\diagdown y}(y,y)},\quad y\in D,\ z\notin D.$$

Applying this formula for x=y, we obtain $\frac{p_{D\diagdown y}(y,z)}{1-p_{D\diagdown y}(y,y)}=p_{D}(y,z)=s_{D}(y,z)=s_{1}(y,z)$ and hence

$$p_{D\diagdown y}(\cdot,z)=s_{1}(\cdot,z)-s_{1}(\cdot,y)s_{1}(y,z), $$

(50)

Using formula (50) for x=z, we obtain p _D╲y (z,z)=s ₁(z,z)−s ₁(z,y)s ₁(y,z), so

$$1-s_{1}(y,z)s_{1}(z,y)A=\bigl(1-s_{1}(z,z)\bigr)A. $$

(51)

Now using this equality and formula (50), formula (49) can be written as

(52)

Using x=y in (52), we obtain s ₂(y,y)=s ₁(y,y)(1−s ₁(z,z))A+s ₁(y,z)s ₁(z,y)A. Then, using formula (51), we obtain 1−s ₂(y,y)=(1−s ₁(y,y))(1−s ₁(z,z))A and dividing both part of formula (52) by 1−s ₂(y,y), we obtain that (48) for y∈D can be written as (47). □