Penalized FTRL with Time-Varying Constraints

Abstract

In this paper we extend the classical Follow-The-Regularized-Leader (FTRL) algorithm to encompass time-varying constraints, through adaptive penalization. We establish sufficient conditions for the proposed Penalized FTRL algorithm to achieve $\mathcal O(\sqrt{t})$ regret and violation with respect to a strong benchmark $\hat{X}^{max}_t$. Lacking prior knowledge of the constraints, this is probably the largest benchmark set that we can reasonably hope for. Our sufficient conditions are necessary in the sense that when they are violated there exist examples where $\mathcal O(\sqrt{t})$ regret and violation is not achieved. Compared to the best existing primal-dual algorithms, Penalized FTRL substantially extends the class of problems for which $\mathcal O(\sqrt{t})$ regret and violation performance is achievable.

Appendix A: Proofs

1.1 A.1 Proof of Lemma 1

Proof

Firstly note that for feasible points $x\in X$ we have that $g^{(j)}(x)\le 0$, $j=1,\cdots ,m$ and so $F(x)=f(x)$. By definition $f(x) \ge f^*=\inf _{x\in X} f(x)$ and so the stated result holds trivially for such points. Now consider an infeasible point $w\notin X$. Let z be an interior point satisfying $g^{(j)}(z)<0$, $j=1,\cdots ,m$; by assumption such a point exists. Let ${\gamma }_0 = \frac{f^*-f(z)-1}{G}$. It is sufficient to show that $F(w)> f^*$ for ${\gamma }\ge {\gamma }_0$ and $G=\max _{j\in \{1,\cdots ,m\}}\{g^{(j)}(z)\}$.

Let $v=\beta z + (1-\beta ) w$ be a point on the chord between points w and z, with $\beta \in (0,1)$ and v on the boundary of X (that is $g^{(j)}(v)\le 0$ for all $j=1,\cdots ,m$ and $g^{(j)}(v)=0$ for at least one $j\in \{1,\cdots ,m\}$). Such a point v exists since z lies in the interior of X and $w\notin X$. Let $A:=\{j:j\in \{1,\cdots ,m\},g^{(j)}(v)=0\}$ and $t(x):=f(x)+{\gamma }\sum _{j\in A} g^{(j)}(x)$. Then $t(v)=f(v)\ge f^*$. Also, by the convexity of $g^{(j)}(\cdot )$ we have that for $j\in A$ that $g^{(j)}(v) = 0 \le \beta g^{(j)}(z) + (1-\beta ) g^{(j)}(w)$. Since $g^{(j)}(z)<0$, it follows that $g^{(j)}(w)>0$. Hence, $\sum _{j\in A}g^{(j)}(w) = \sum _{j\in A}\max \{0,g^{(j)}(w)\} \le \sum _{j=1}^m\max \{0,g^{(j)}(w)\}$ and so $t(w) \le F(w,{\gamma })$. Now, observe that $t(z)= f(z)+{\gamma }\sum _{j\in A} g^{(j)}(z) \le f(z)+{\gamma }_0\sum _{j\in A} g^{(j)}(z)$ since $g^{(j)}(z)<0$ and ${\gamma }\ge {\gamma }_0$. Hence,

$$\begin{aligned} t(z)&\le f(z)+(f^*-f(z)-1)\frac{\sum _{j\in A} g^{(j)}(z)}{G} \end{aligned}$$

(8)

Selecting G such that $\frac{\sum _{j\in A} g^{(j)}(z)}{G}\ge 1$ then $t(z) \le f^*-1 \le t(v) -1$. So we have established that $f^*\le t(v)$, $t(z)\le t(v)-1$ and $t(w) \le F(w)$. Finally, by the convexity of $t(\cdot )$, $t(v) \le \beta t(z) + (1-\beta ) t(w)$. Since $t(z)\le t(v)-1$ it follows that $t(v) \le \beta (t(v)-1) + (1-\beta ) t(w)$ i.e. $t(v) \le -\frac{\beta }{1-\beta }+t(w)$. Therefore $f^* \le -\frac{\beta }{1-\beta } + F(w)<F(w)$ as claimed.