A monotonic policy optimization algorithm for high-dimensional continuous control problem in 3D MuJoCo

Abstract

One challenge in applying reinforcement learning with nonlinear function approximator to high- dimensional continuous control problems is that the update policy produced by the many existed algorithms may fail to improve policy performance or even causes a serious degradation of the policy performance. To address this challenge, this paper proposes a new lower bound on the policy improvement where an average policy divergence on state space is penalized. To the best of our knowledge, this is currently the best result about the lower bound on the policy improvement. Optimizing directly the lower bound on the policy improvement is very difficult, because it demands for high computational overhead. According to the ideal of the trust region policy optimization (TRPO), this paper also presents a monotonic policy optimization algorithm, which is based on the new lower bound on the policy improvement introduced in this paper, it can generate a sequence of monotonically improving policies, and it is suitable for the large-scale continuous control problems. This paper also evaluates and compares the proposed algorithms with some of the existed algorithms on highly challenging robot locomotion tasks.

Appendix

• A. Proof of Theorem 1

Proof: for convenience, define

$$ {\delta}_f\left(s,a,{s}^{\prime}\right)=R\left(s,a\right)+\gamma f(s)-f\left({s}^{\prime}\right),\kern0.5em {\overline{\delta}}_f^{\pi^{\prime }}(s)=\underset{a\sim {\pi}^{\prime },{s}^{\prime}\sim P}{E}\left({\delta}_f\left(s,a,{s}^{\prime}\right)\right)\left[s\right], $$

(22)

where, s’ is next state given that the agent took action a in state s, by Lemma 1, the identity is obtained

$$ {J}_{\mu}^{\pi^{\prime }}-{J}_{\mu}^{\pi }=\frac{1}{1\hbox{-} \gamma}\left(\underset{s\sim {d}_{\mu}^{\pi^{\prime }},a\sim {\pi}^{\prime },{s}^{\prime}\sim P}{E}\left[{\delta}_f\left(s,a,{s}^{\prime}\right)\right]-\underset{s\sim {d}_{\mu}^{\pi },a\sim \pi, {s}^{\prime}\sim P}{E}\left[{\delta}_f\right(s,a,{s}^{\prime}\left)\right]\right) $$

(23)

$\underset{s\sim {d}_{\mu}^{\pi^{\prime }},a\sim {\pi}^{\prime },{s}^{\prime}\sim P}{E}\left[{\delta}_f\left(s,a,{s}^{\prime}\right)\right]$ is rewritten in the inner product form as following

$$ {\displaystyle \begin{array}{l}\underset{s\sim {d}_{\mu}^{\pi^{\prime }},a\sim {\pi}^{\prime },{s}^{\prime}\sim P}{E}\left({\delta}_f\left(s,a,{s}^{\prime}\right)\right)=\left\langle {d}_{\mu}^{\pi^{\prime }},{\overline{\delta}}_f^{\pi^{\prime }}\right\rangle =\left\langle {d}_{\mu}^{\pi },{\overline{\delta}}_f^{\pi^{\prime }}\right\rangle +\left\langle {d}_{\mu}^{\pi^{\prime }}-{d}_{\mu}^{\pi },{\overline{\delta}}_f^{\pi^{\prime }}\right\rangle \\ {}\ge \left\langle {d}_{\mu}^{\pi },{\overline{\delta}}_f^{\pi^{\prime }}\right\rangle -{\left\Vert {d}_{\mu}^{\pi^{\prime }}-{d}_{\mu}^{\pi}\right\Vert}_1\frac{\Delta {\overline{\delta}}_f^{\pi^{\prime }}}{2}\kern0.5em \left(\mathrm{by}\ \mathrm{Lemma}\ 3\right)\end{array}} $$

(24)

Different inequalities applied for dealing with the inner product $\left\langle {d}^{\pi^{\prime }}-{d}^{\pi },{\overline{\delta}}_f^{\pi^{\prime }}\right\rangle$ may lead to different lower bounds, such as Holder’s inequality [1]. Here we use the Lemma 3. To our knowledge, the lower bound here derived by us is better than the others. By the importance sampling, the identity can be rewritten as follows

$$ \left\langle {d}_{\mu}^{\pi },{\overline{\delta}}_f^{\pi^{\prime }}\right\rangle =\underset{s\sim {d}_{\mu}^{\pi^{\prime }},a\sim {\pi}^{\prime },{s}^{\prime}\sim P}{E}\left[{\delta}_f\left(s,a,{s}^{\prime}\right)\right]=\underset{s\sim {d}_{\mu}^{\pi },a\sim \pi, {s}^{\prime}\sim P}{E}\left[\frac{\pi^{\prime}\left(a|s\right)}{\pi \left(a|s\right)}\left(R\left(s,a\right)+\gamma f\left({s}^{\prime}\right)-f(s)\right)\right] $$

(25)

Bring all terms together, the lower bound of the policy improvement can be obtained.

• B. Proof of Corollary 1

Proof: let

$$ f(s)={V}^{\pi }(s),\kern0.5em {\delta}_f\left(s,a,{s}^{\prime}\right)={\delta}_V\left(s,a,{s}^{\prime}\right)=R\left(s,a\right)+\gamma {V}^{\pi}\left({s}^{\prime}\right)-{V}^{\pi }(s) $$

(26)

(TD -error), obviously,

$$ \underset{s\sim {d}_{\mu}^{\pi },a\sim \pi, {s}^{\prime}\sim P}{E}\left[{\delta}_{V^{\pi }}\left(s,a,{s}^{\prime}\right)\right]=\underset{s\sim {d}_{\mu}^{\pi }}{E}\left[{A}^{\pi}\left(s,a\right)\right]=0, $$

(27)

$$ \left\langle {d}_{\mu}^{\pi },{\overline{\delta}}_f^{\pi^{\prime }}\right\rangle =\underset{s\sim {d}_{\mu}^{\pi },a\sim {\pi}^{\prime },{s}^{\prime}\sim P}{E}\left[{\delta}_f\left(s,a,{s}^{\prime}\right)\right]=\underset{s\sim {d}_{\mu}^{\pi },a\sim {\pi}^{\prime }}{E}\left[\frac{\pi^{\prime}\left(a,s\right)}{\pi \left(a,s\right)}{A}^{\pi}\left(s,a\right)\right] $$

(28)

$$ \Delta {\overline{\delta}}_V^{\pi^{\prime }}={\max}_{i,j}\left|{\overline{\delta}}_V^{\pi^{\prime }}(i)-{\overline{\delta}}_V^{\pi^{\prime }}(j)\right|={\max}_{i,j}\left|\underset{a\sim {\pi}^{\prime }}{E}\left[{A}^{\pi}\left(i,a\right)\right]-\underset{a\sim {\pi}^{\prime }}{E}\left[{A}^{\pi}\right(j,a\left)\right]\right|={\max}_{i,j}\left|\underset{a\sim {\pi}^{\prime }}{E}\left[{A}^{\pi}\left(i,a\right)\hbox{-} {A}^{\pi}\right(j,a\left)\right]\right| $$

(29)

by Theorem 1, Eq. (13) is obtained.