Reinforcement learning algorithms that combine policy and value estimation
The actor-critic algorithm (AC) is a family of reinforcement learning (RL) algorithms that combine policy-based RL algorithms such as policy gradient methods, and value-based RL algorithms such as value iteration, Q-learning, SARSA, and TD learning.
An AC algorithm consists of two main components: an "actor" that determines which actions to take according to a policy function, and a "critic" that evaluates those actions according to a value function. Some AC algorithms are on-policy, some are off-policy. Some apply to either continuous or discrete action spaces. Some work in both cases.
Overview.
The actor-critic methods can be understood as an improvement over pure policy gradient methods like REINFORCE via introducing a baseline.
Actor.
The actor uses a policy function formula_1, while the critic estimates either the value function formula_2, the action-value Q-function formula_3, the advantage function formula_4, or any combination thereof.
The actor is a parameterized function formula_5, where formula_6 are the parameters of the actor. The actor takes as argument the state of the environment formula_7 and produces a probability distribution formula_8.
If the action space is discrete, then formula_9. If the action space is continuous, then formula_10.
The goal of policy optimization is to improve the actor. That is, to find some formula_6 that maximizes the expected episodic reward formula_12:formula_13where formula_14 is the discount factor, formula_15 is the reward at step formula_16, and formula_17 is the time-horizon (which can be infinite).
The goal of policy gradient method is to optimize formula_12 by gradient ascent on the policy gradient formula_19.
As detailed on the policy gradient method page, there are many unbiased estimators of the policy gradient:formula_20where formula_21 is a linear sum of the following:
Critic.
In the unbiased estimators given above, certain functions such as formula_32 appear. These are approximated by the critic. Since these functions all depend on the actor, the critic must learn alongside the actor. The critic is learned by value-based RL algorithms.
For example, if the critic is estimating the state-value function formula_33, then it can be learned by any value function approximation method. Let the critic be a function approximator formula_34 with parameters formula_35.
The simplest example is TD(1) learning, which trains the critic to minimize the TD(1) error:formula_36The critic parameters are updated by gradient descent on the squared TD error:formula_37where formula_38 is the learning rate. Note that the gradient is taken with respect to the formula_35 in formula_40 only, since the formula_35 in formula_42 constitutes a moving target, and the gradient is not taken with respect to that. This is a common source of error in implementations that use automatic differentiation, and requires "stopping the gradient" at that point.
Similarly, if the critic is estimating the action-value function formula_43, then it can be learned by Q-learning or SARSA. In SARSA, the critic maintains an estimate of the Q-function, parameterized by formula_35, denoted as formula_45. The temporal difference error is then calculated as formula_46. The critic is then updated byformula_47The advantage critic can be trained by training both a Q-function formula_48 and a state-value function formula_34, then let formula_50. Although, it is more common to train just a state-value function formula_34, then estimate the advantage byformula_52Here, formula_53 is a positive integer. The higher formula_53 is, the more lower is the bias in the advantage estimation, but at the price of higher variance.
The Generalized Advantage Estimation (GAE) introduces a hyperparameter formula_55 that smoothly interpolates between Monte Carlo returns (formula_56, high variance, no bias) and 1-step TD learning (formula_57, low variance, high bias). This hyperparameter can be adjusted to pick the optimal bias-variance trade-off in advantage estimation. It uses an exponentially decaying average of n-step returns with formula_55 being the decay strength.
References.
 ;templatestyles src="Reflist/styles.css" / ;