Four proofs of Gittins’ multiarmed bandit theorem

Abstract

We study four proofs that the Gittins index priority rule is optimal for alternative bandit processes. These include Gittins’ original exchange argument, Weber’s prevailing charge argument, Whittle’s Lagrangian dual approach, and Bertsimas and Niño-Mora’s proof based on the achievable region approach and generalized conservation laws. We extend the achievable region proof to infinite countable state spaces, by using infinite dimensional linear programming theory.

Appendix: Proofs of index properties

In this appendix we present the proofs of some results postponed in the paper.

Proof of Theorem 2.1

The direct proof is quite straightforward. Note that in (2.1) the value of ν(i,σ) for each stopping time σ is a ratio of sums over consecutive times. Hence to compare these we use the simple inequality:

$$ \frac{a}{c}<\frac{a+b}{c+d} \quad \Longleftrightarrow \quad \frac{a+b}{c+d}<\frac{b}{d} \quad \Longleftrightarrow\quad \frac{a}{c}<\frac{b}{d} $$

(6.1)

We show that if we start from i and wish to maximize ν(i,σ) it is never optimal to stop in state j if ν(j)>ν(i) or to continue in state j if ν(j)<ν(i), which allows us to consider only stopping time of the form (2.3). We then assume that no stopping time achieves the supremum, and we construct an increasing sequence of stopping times with increasing ratio, which converges to τ(i) which therefore must achieve the supremum, and from this contradiction we deduce that the supremum is achieved. Finally we show that since the supremum is achieved, it is achieved by τ(i) as well as by all stopping times of the form (2.3).

Step 1: Any stopping time which stops while the ratio is >ν(Z(0)) does not achieve the supremum. Assume that Z(0)=i, fix j such that ν(j)>ν(i), and consider a stopping time σ such that:

$$ \mathbb{P}\bigl(Z(\sigma) = j\bigm|Z(0)=i\bigr)>0. $$

(6.2)

By the definition (2.1) there exists a stopping time σ′ such that $\nu(j,\sigma ')> \frac{\nu(j)+\nu(i)}{2}$. Define σ′=0 for all initial values ≠j. Then:

$$\begin{aligned} & \nu\bigl(i,\sigma + \sigma '\bigr) \\ &\quad= \frac{\mathbb{E}\{ \sum_{t=0}^{\sigma-1} \alpha^t R(Z(t)) \mid Z(0)=i \} + \mathbb{E}\{ \sum_{t=\sigma}^{\sigma + \sigma '-1} \alpha^t R(Z(t)) \mid Z(0)=i \} }{ \mathbb{E}\{ \sum_{t=0}^{\sigma-1} \alpha^t \mid Z(0)=i \} + \mathbb{E}\{ \sum_{t=\sigma}^{\sigma+ \sigma '-1} \alpha^t \mid Z(0)=i \} } \\ &\quad > \nu(i,\sigma), \end{aligned}$$

by (6.1), (6.2).

Step 2: Any stopping time which continues when the ratio is <ν(Z(0)) does not achieve the supremum. Assume that Z(0) = i, fix j such that ν(j) < ν(i), and let σ′ = min{t:Z(t) = j}. Consider any stopping time σ which does not always stop when it reaches state j, and assume that:

$$ \nu(i,\sigma)>\nu(j) \quad\mbox{and}\quad \mathbb{P}\bigl(\sigma > \sigma '\mid Z(0)=i\bigr)>0. $$

(6.3)

Then:

$$\begin{aligned} \nu(i,\sigma) = & \frac{\mathbb{E}\{ \sum_{t=0}^{\min(\sigma,\sigma ')-1} \alpha^t R(Z(t)) \mid Z(0)=i \} + \mathbb{E}\{ \sum_{t=\sigma '}^{\sigma-1} \alpha^t R(Z(t)) \mid Z(\sigma ')=j \} }{ \mathbb{E}\{ \sum_{t=0}^{\min(\sigma,\sigma ')-1} \alpha^t \mid Z(0)=i \} + \mathbb{E}\{ \sum_{t=\sigma '}^{\sigma-1} \alpha^t \mid Z(\sigma ')=j \} } \\ <& \nu\bigl(i,\min\bigl(\sigma,\sigma '\bigr)\bigr), \end{aligned}$$

by (6.1), (6.3).

Steps 1, 2 show that the supremum can be taken over stopping times σ>0 which satisfy (2.3), and we restrict attention to such stopping times only.

Step 3: The supremum is achieved. If τ(i) is the unique stopping time which satisfies (2.3) then it achieves the supremum and there is nothing more to prove. Assume that the supremum is not achieved. We now consider a fixed stopping time σ>0 which satisfies (2.3) and:

$$ \mathbb{P}\bigl(\sigma < \tau(i)\mid Z(0)=i\bigr) > 0, \quad \nu(i, \sigma)=\nu_0<\nu(i). $$

(6.4)

This is possible, since τ is not unique, and since the supremum is not achieved. Assume that σ stops at a time <τ(i) when the state is Z(σ)=j. By (2.3), ν(j)=ν(i). We can then find σ′ such that $\nu(j,\sigma ') \ge \frac{\nu_{0}+\nu(i)}{2}$. Define σ′ accordingly for the value of Z(σ) whenever σ<τ(i), and let σ′=0 if σ=τ(i). Let σ ₁=σ+σ′. Clearly we have (repeat the argument of step 1):

$$ \sigma \le \sigma_1 \le \tau(i), \quad \nu(i, \sigma) < \nu(i,\sigma_1)=\nu_1<\nu(i) $$

(6.5)

We can now construct a sequence of stopping times, with

$$ \sigma_{n-1} \le \sigma_n \le \tau(i), \quad \nu(i,\sigma_{n-1}) < \nu(i,\sigma_n)= \nu_n<\nu(i) $$

(6.6)

which will continue indefinitely, or will reach $\mathbb{P}(\sigma_{n_{0}} = \tau(i))=1$, in which case we define σ _n =τ(i),n>n ₀.

It is easy to see that min(n,σ _n )=min(n,τ(i)), hence σ _n ↗τ(i) a.s. It is then easy to see (use dominated or monotone convergence) that ν(i,σ _n )↗ν(i,τ(i)). But this implies that ν(i,σ)<ν(i,τ(i)). Hence the assumption that the supremum is not achieved implies that the supremum is achieved by τ(i), which is a contradiction. Hence, for any initial state Z(0)=i the supremum is achieved by some stopping time, which satisfies (2.3).

Step 4: The supremum is achieved by τ(i). Start from Z(0)=i, and assume that a stopping time σ satisfies (2.3) and achieves the supremum. Assume

$$ \mathbb{P}\bigl(\sigma < \tau(i)\mid Z(0)=i\bigr) > 0, \quad \nu(i, \sigma)=\nu(i) $$

(6.7)

and take the event that σ stops at a time <τ(i) when the state is Z(σ)=j. By (2.3) ν(j)=ν(i). We can then find σ′ which achieves the supremum, ν(j,σ′)=ν(j)=ν(i). Define σ′ accordingly for the value of Z(σ) whenever σ<τ(i), and let σ′=0 if σ=τ(i). Let σ ₁=σ+σ′. Clearly we have:

$$ \sigma \le \sigma_1 \le \tau(i), \quad \nu(i, \sigma) = \nu(i,\sigma_1)= \nu(i) $$

(6.8)

We can now construct an increasing sequence of stopping times, σ _n ↗τ(i) a.s., and all achieving ν(i,σ _n )=ν(i). Hence (again use dominated or monotone convergence) ν(i,τ(i))=ν(i).

Step 5: The supremum is achieved by any stopping time which satisfies ( 2.3 ). Let σ satisfy (2.3). Whenever σ<τ(i) and Z(σ)=j, we will have τ(i)−σ=τ(j), and ν(j,τ(i)−σ)=ν(i). Hence:

$$\begin{aligned} \nu(i) = \nu\bigl(i,\tau(i)\bigr) = & \frac{\mathbb{E}\{ \sum_{t=0}^{\sigma-1} \alpha^t R(Z(t)) \mid Z(0)=i \} + \mathbb{E}\{ \sum_{t=\sigma}^{\tau(i)-1} \alpha^t R(Z(t)) \mid Z(0)=i \} }{ \mathbb{E}\{ \sum_{t=0}^{\sigma-1} \alpha^t \mid Z(0)=i \} + \mathbb{E}\{ \sum_{t=\sigma}^{\tau(i)-1} \alpha^t \mid Z(0)=i \} } \\ = & \nu(i,\sigma). \end{aligned}$$

This completes the proof. □

Proof of Proposition 2.2

Step 1: We show that ν(i)≤γ(i). Consider any y<ν(i), let $M=\frac{y}{1-\alpha}$. By definition (2.1) there exists a stopping time τ for which ν(i,τ)>y.

Hence, a policy π which from state i will play up to time τ and then stop and collect the reward M, will have:

$$\begin{aligned} V_r^\pi(i,M) =& \mathbb{E}\Biggl\{ \sum _{t=0}^{\tau-1} \alpha^t R\bigl(Z(t)\bigr) + \sum_{t=\tau}^{\infty} \alpha^t y \biggm| Z(0)=i \Biggr\} \\ > & \mathbb{E}\Biggl\{ \sum_{t=0}^{\tau-1} \alpha^t y + \sum_{t=\tau}^{\infty} \alpha^t y \biggm| Z(0)=i \Biggr\} = \frac{y}{1-\alpha} = M. \end{aligned}$$

Hence V _r (i,M)>M, and i belongs to the continuation set, for standard arm reward y, (or fixed charge y, or terminal reward M). Hence, M(i)≥M, and γ(i)≥y. But y<ν(i) was arbitrary. Hence, γ(i)≥ν(i).

Step 2: We show that ν(i)≥γ(i). Consider any y<γ(i). Let $M=\frac{y}{1-\alpha}$, and consider τ(i,M) and V _r (i,M). Writing (2.14), and using the fact that for M<M(i) we have i∈C _M and V _r (i,M)>M:

$$\begin{aligned} V_r(i,M) =& \mathbb{E}\Biggl\{ \sum_{t=0}^{\tau(i,M)-1} \alpha^t R\bigl(Z(t)\bigr) + \sum_{t=\tau(i,M)}^{\infty} \alpha^t y \biggm| Z(0)=i \Biggr\} \\ >& \frac{y}{1-\alpha}. \end{aligned}$$

But this means that ν(i,τ(i,M))>y. Hence, ν(i)>y. But y<γ(i) was arbitrary. Hence, ν(i)≥γ(i).

Step 3: Identification of lim _m↗M(i) τ(i;m) with τ(i) in ( 2.1 ). Clearly, starting from state i, τ(i;m) with m<M(i) will continue for all states j with M(j)≥M(i), and for every state j with M(j)<M(i) it will retire at state j if M(j)<m<M(i), so lim _m↗M(i) τ(i;m)=τ(i;M(i))=τ(i). The last equality holds since we showed that $\frac{M(i)}{1-\alpha}=\gamma(i)=\nu(i)$. □

Proof of Proposition 2.9

The equivalence of the two algorithms is easily seen. Step 1 is identical. Assume that the two algorithms are the same for steps 1,…,k−1. We then have in step k, for any $i \in S_{\varphi(k-1)}^{-}$ that:

$$\begin{aligned} & \frac{R(i) - \sum_{j=1}^{k-1} A_i^{S_{\varphi(j)}} y^{S_{\varphi(j)}} }{A_i^{S_{\varphi(k-1)}^-}} \\ &\quad= \frac{R(i) - A_i^{S_{\varphi(1)}} \nu(\varphi(1)) - \sum_{j=2}^{k-1} A_i^{S_{\varphi(j)}} (\nu(\varphi(j)) - \nu(\varphi(j-1))) }{ A_i^{S_{\varphi(k-1)}^-}} \\ &\quad= \frac{R(i) - \sum_{j=1}^{k-1} A_i^{S_{\varphi(j)}} \nu(\varphi(j)) + \sum_{j=1}^{k-2} A_i^{S_{\varphi(j+1)}} \nu(\varphi(j)) }{ A_i^{S_{\varphi(k-1)}^-}} \\ &\quad = \frac{R(i) + \sum_{j=1}^{k-1} (A_i^{S_{\varphi(j)}^-}-A_i^{S_{\varphi(j)}}) \nu(\varphi(j)) - A_i^{S_{\varphi(k-1)}^-} \nu(\varphi(k-1))}{ A_i^{S_{\varphi(k-1)}^-}} \\ &\quad = \frac{R(i) + \sum_{j=1}^{k-1} (A_i^{S_{\varphi(j)}^-}-A_i^{S_{\varphi(j)}}) \nu(\varphi(j)) }{ A_i^{S_{\varphi(k-1)}^-}} - \nu\bigl(\varphi(k-1)\bigr) \end{aligned}$$

and so the supremum in step k is achieved by the same φ(k) in both versions of the algorithm. The quantities y ^S appear in the 4th proof, Sect. 3.4. □

Proof of Proposition 4.2

(i) The definition of Ax(S) implies that Ax(∅)=0 because it is an empty sum. Trivially $T_{j}^{\emptyset}=\infty$, so $\alpha^{T_{j}^{\emptyset}}=0$, hence the definition (3.20) of b(S) implies b(∅)=0. Thus (i) holds.

(ii) First we show that if x∈X, and y(S)=Ax(S), then g _y is of bounded variation. Let $x^{+}=\{x_{i}^{+} = \max(x_{i},0)\}_{i\in E}$, and $x^{-}=\{x_{i}^{-} = \max(-x_{i},0)\}_{i\in E}$. Clearly x ⁺,x ⁻∈X and g _y (v)=Ax ⁺(S(v))−Ax ⁻(S(v)). We have:

$$\begin{aligned} Ax^{+}\bigl(S(v)\bigr) =&\sum _{j \in S(v)}A_{j}^{S(v)}x^{+}_{j} \\ =&\sum_{j \in E}A_{j}^{S(v)}x^{+}_{j}- \sum_{j \in E\backslash S(v)}A_{j}^{S(v)}x^{+}_{j} \end{aligned}$$

(6.9)

S(v) is increasing in v, as a result $A_{j}^{S(v)}$ is decreasing in v. Also E∖S(v) is decreasing in v. Hence both terms in (6.9) decrease in v. Thus Ax ⁺(S(v)), as the difference of two decreasing functions of v is of bounded variation. Similarly Ax ⁻(S(v)), is of bounded variation. Hence, g _y is of bounded variation.

Next we take y=b, and consider g _y . g _y (v)=b(S(v)) increases in v thus it is of bounded variation. This proves (ii).

(iii) To calculate the limits from the left we need to prove some continuity results: consider an increasing sequence {v _n }, such that lim _n→∞ v _n =v, and v _n <v. Consider first S(v _n ) and S ⁻(v). Then S(v _n ) are increasing and $\lim_{n\to \infty} S(v_{n}) = \bigcup_{n=0}^{\infty} S(v_{n}) = S^{-}(v)$. To see this, note the if i∈S ⁻(v) then ν(i)<v, hence for some n ₀ we have ν(i)≤v _n for all n≥n ₀, hence i∈S(v _n ),n≥n ₀.

Next consider $T_{j}^{S(v_{n})}$ and $T_{j}^{S^{-}(v)}$ for some given sample path. If $T_{j}^{S^{-}(v)}= \infty$ then $T_{j}^{S(v_{n})}= \infty$ for all n. If $T_{j}^{S^{-}(v)}= t < \infty$ then we have Z(t)=i∈S ⁻(v), but in that case i∈S(v _n ),n>n ₀, so $T_{j}^{S(v_{n})}=t$ for all N≥n ₀. Thus we have that $T_{j}^{S(v_{n})}$ is non-increasing in n and converges to $T_{j}^{S^{-}(v)}$ for this sample path. Hence $T_{j}^{S(v_{n})} \searrow_{\rm a.s.} T_{j}^{S^{-}(v)}$.

We now have that $\sum_{t=0}^{T_{j}^{S(v_{n})}-1} \alpha^{t} \searrow_{\rm a.s.} \sum_{t=0}^{T_{j}^{S^{-}(v)}-1}\alpha^{t}$, and because ∑ _t α ^t are uniformly bounded, $A_{j}^{S(v_{n})} \searrow A_{j}^{S^{-}(v)}$.

Consider now Ax(S(v _n )) and Ax(S ⁻(v)). We need to show that as n→∞,

$$\sum_{j \in S(v_n)}A_{j}^{S(v_n)}x_j - \sum_{j \in S^-(v)}A_{j}^{S^-(v)} x_j = \sum_{j \in S(v_n)} \bigl( A_{j}^{S(v_n)} - A_{j}^{S^-(v)} \bigr) x_j - \sum_{j \in S^-(v) \backslash S(v_n)} A_{j}^{S^-(v)} x_j \to 0 $$

which follows from $A_{i}^{S}$ bounded by $\frac{1}{1-\alpha}$, x _i absolutely convergent, and S(v _n )↗S ⁻(v). To explain this a little further: Since x _j are absolutely convergent, for every ϵ>0 we can find a finite subset of states E ₀ such that $\frac{2}{1-\alpha} \sum_{j\in E\backslash E_{0}} |x_{j}| < \frac {1}{2}\epsilon$. If we now examine the sums only over j∈E ₀, clearly the first sum can be made arbitrarily small as n→∞, and the second sum becomes empty as n→∞.

Finally for any given Z(0), $\mathbb{E}(\alpha^{T_{Z_{k}(0)}^{S(v_{n})}}) \nearrow \mathbb{E}(\alpha^{T_{Z_{k}(0)}^{S^{-}(v)}}),k=1,\ldots,N$ and hence by definition (3.20) b(S(v _n ))↗b(S ⁻(v)). This completes the proof that g _y (v _n )→y(S ⁻(v)).

To show continuity from the right, consider a decreasing sequence {v _n }, such that lim _n→∞ v _n =v, and v _n >v. Consider the sequence of sets S(v _n ) and the set S(v). Then S(v _n ) are decreasing and $\lim_{n\to \infty} S(v_{n}) = \bigcap_{n=0}^{\infty} S(v_{n}) = S(v)$. To see this, note the if i∉S(v) then ν(i)>v, hence for some n ₀ we have ν(i)>v _n for all n≥n ₀, hence i∉S(v _n ),n≥n ₀.

The remaining steps of the proof are as for the limit from the left: one shows that $T_{j}^{S(v_{n})} \nearrow_{\rm a.s.} T_{j}^{S(v)}$, and so on. This completes the proof of (iii).

(iv) We wish to show

$$y\bigl(S(v)\bigr) - y\bigl(S^-(v)\bigr) = \sum_{i:\nu(i)=v} y(S_i) - y\bigl(S^-_i\bigr), $$

Clearly, if {i:ν(i)=v}=∅ then both sides are 0, and if {i:ν(i)=v} consists of a single state {i} then $S(v)=S_{i}, S^{-}(v)=S^{-}_{i}$ and there is nothing to prove. If {i:ν(i)=v} consists of a finite set of states with i ₁≺i ₂…≺i _M , then S(i _k )=S ⁻(i _k+1) and the summation over i _k is a collapsing sum. If {i:ν(i)=v} is infinite countable but well ordered, then we can order {i:ν(i)=v} as i ₁≺i ₂≺… (ordinal type ω), and the infinite sum on the right is a collapsing sum, which converges to the right hand side.

The difficulty here is that in the general case {i:ν(i)=v} may not be well ordered by ≺, and it is this general case which we wish to prove. We do so by a sample path argument, using the fact that the sequence of activation times, t=1,2,… is well ordered. In our sample path argument we make use of the sequence of stopping times and states ${\mathcal {T}}_{\ell}$ and k _ℓ , defined in (2.21). Recall that this is the sequence of stopping times and states at which the index sample path ‘loses priority height’, in that at time ${\mathcal {T}}_{\ell}$ it reaches a state k _ℓ which is of lower priority than all the states encountered in $0<t<{\mathcal {T}}_{\ell}$.

Fix a value of v, and consider a sample path starting from Z(0)=j. Then for this sample path we will have integers $0 \le \underline{L} < \overline{L} \le \infty$ such that:

$$\nu(k_\ell)= \left \{\begin{array}{l@{\quad}l} > v & \ell \le \underline{L} \\ = v & \underline{L} < \ell < \overline{L} \\ < v & \ell \ge \overline{L} \end{array} \right . $$

It is possible that $\underline{L} = \infty$ because the sample path never reaches S(v). It is also possible that $\overline{L} = \underline{L}+1$, either because {i:ν(i)=v}=∅, or because the first visit of the sample path to S(v) is directly to a state with index <v; in each of these cases $T_{j}^{S(v)} = T_{j}^{S^{-}(v)}$. Otherwise, if $\overline{L} - \underline{L} > 1$, then ${\mathcal {T}}_{\underline{L}+1}= T_{j}^{S_{k_{\underline{L}+1}}}=T_{j}^{S(v)}$ will be the first visit of the process in S(v), and ${\mathcal {T}}_{\overline{L}}= T_{j}^{S^{-}_{k_{\overline{L}-1}}}=T_{j}^{S^{-}(v)}$ will be the first visit of the process in S ⁻(v).

We can then write:

$$ \sum_{t=0}^{T_j^{S^-(v)} - 1} \alpha^t - \sum_{t=0}^{T_j^{S(v)}-1} \alpha^t = \sum_{\underline{L} < \ell < \overline{L}} \Biggl( \sum _{t=0}^{T_j^{S^-_{k_\ell}} - 1} \alpha^t - \sum _{t=0}^{T_j^{S_{k_\ell}} - 1} \alpha^t \Biggr) = \sum_{i:\nu(i)=v} \Biggl( \sum_{t=0}^{T_j^{S^-_i} - 1} \alpha^t - \sum_{t=0}^{T_j^{S_i} - 1} \alpha^t \Biggr) $$

(6.10)

where the second equality follows from $T_{j}^{S^{-}_{i}}=T_{j}^{S_{i}}$ for all i:ν(i)=v except for $k_{\ell}, \underline{L} < \ell < \overline{L}$.

By taking expectations we now get that:

$$A_j^{S^-(v)} - A_j^{S(v)} = \sum _{i:\nu(i)=v} A_j^{S^-_i} - A_j^{S_i} $$

and also, for all j∈S(v)∖S ⁻(v):

$$A_j^{S^-_j} - A_j^{S(v)} = \sum _{ i:\nu(i)=v, i \succeq j} \bigl( A_j^{S^-_i} - A_j^{S_i} \bigr). $$

Next we note that

$$b(S)=\mathbb{E}\biggl(\frac{\alpha^{T_{\mathbf{Z}(0)}^S}}{1-\alpha}\biggr) =\mathbb{E}\Biggl(\sum _{t=T_{\mathbf{Z}(0)}^S}^\infty \alpha^t\Biggr) $$

where $T_{\mathbf{Z}(0)}^{S} = \sum_{n=1}^{N} T_{Z_{n}(0)}^{S}$. Hence, using the same sample path result (6.10) for each of Z _n (0) we get

$$b\bigl(S(v)\bigr)-b\bigl(S^-(v)\bigr)= \mathbb{E}\Biggl(\sum _{t=T_{\mathbf{Z}(0)}^{S(v)}}^{T_{\mathbf{Z}(0)}^{S^-(v)}-1} \alpha^t\Biggr) = \mathbb{E}\Biggl(\sum_{i:\nu(i)=v} \sum_{t=T_{\mathbf{Z}(0)}^{S_i}}^{T_{\mathbf{Z}(0)}^{S^-_i}-1} \alpha^t\Biggr) = \sum_{i:\nu(i)=v} \bigl( b(S_i)-b\bigl(S^-_i\bigr) \bigr) $$

We now come to evaluate Ax(S ⁻(v))−Ax(S(v)). We need to show that for all x∈X:

$$\begin{aligned} Ax\bigl(S^-(v)\bigr)-Ax\bigl(S(v)\bigr) =& \sum_{j\in S^-(v)} x_j A_j^{S^-(v)} - \sum _{j\in S(v)} x_j A_j^{S(v)} \\ = & \sum_{i:\nu(i)=v} \biggl( \sum _{j\in S^-_i} x_j A_j^{S^-_i} - \sum_{j\in S_i} x_j A_j^{S_i} \biggr) = \sum_{i:\nu(i)=v} \bigl( Ax\bigl(S^-_i \bigr)-Ax(S_i) \bigr) \end{aligned}$$

(6.11)

Since this has to hold for all x, we need to show for every j∈E that the coefficients of x _j satisfy the equalities individually.

For j∈S ⁻(v) clearly $j\in S(v), S_{i}, S^{-}_{i}, i:\nu(i)=v$, we need to check that:

$$A_j^{S^-(v)} - A_j^{S(v)} = \sum _{i:\nu(i)=v} \bigl( A_j^{S^-_i} - A_j^{S_i} \bigr) $$

which we have just shown.

For j∈S(v)∖S ⁻(v) we have: j∈S(v), and for i:ν(i)=v: j∈S _i if i⪰j, and $j\in S^{-}_{i}$ if i≻j. Hence, the coefficients of j which we need to compare are:

$$- A_j^{S(v)} = \sum_{ i:\nu(i)=v, i \succ j} A_j^{S^-_i} - \sum_{i:\nu(i)=v, i \succeq j} A_j^{S_i} $$

Which amounts to:

$$A_j^{S^-_j} - A_j^{S(v)} = \sum _{ i:\nu(i)=v, i \succeq j} \bigl( A_j^{S^-_i} - A_j^{S_i} \bigr) $$

which we have also shown. □