Actor-critic algorithms
V Konda, J Tsitsiklis - Advances in neural information …, 1999 - proceedings.neurips.cc
Advances in neural information processing systems, 1999•proceedings.neurips.cc
We propose and analyze a class of actor-critic algorithms for simulation-based optimization
of a Markov decision process over a parameterized family of randomized stationary policies.
These are two-time-scale algorithms in which the critic uses TD learning with a linear
approximation architecture and the actor is updated in an approximate gradient direction
based on information pro (cid: 173) vided by the critic. We show that the features for the critic
should span a subspace prescribed by the choice of parameterization of the actor. We …
of a Markov decision process over a parameterized family of randomized stationary policies.
These are two-time-scale algorithms in which the critic uses TD learning with a linear
approximation architecture and the actor is updated in an approximate gradient direction
based on information pro (cid: 173) vided by the critic. We show that the features for the critic
should span a subspace prescribed by the choice of parameterization of the actor. We …
Abstract
We propose and analyze a class of actor-critic algorithms for simulation-based optimization of a Markov decision process over a parameterized family of randomized stationary policies. These are two-time-scale algorithms in which the critic uses TD learning with a linear approximation architecture and the actor is updated in an approximate gradient direction based on information pro (cid: 173) vided by the critic. We show that the features for the critic should span a subspace prescribed by the choice of parameterization of the actor. We conclude by discussing convergence properties and some open problems.
proceedings.neurips.cc