<p>The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive nudge.</p>

<p>One way to estimate the baseline is to have a value function approximator. At every step, you train a NN, using the trajectories collected via the current policy, to predict the value function for states.  </p>

<p>I hope that answers your query.</p>

CC BY-SA 4.0
2020-05-20T22:03:53.197Z
2020-05-21T17:22:03.080Z
2020-05-21T17:22:03.080Z
pecey
<p>First let us note the definition of the advantage function: </p>

<p><span class="math-container">$$A(s,a) = Q(s,a) - V(s) \; ,$$</span></p>

<p>where <span class="math-container">$Q(s,a)$</span> is the action-value function and <span class="math-container">$V(s)$</span> is the state-value function. In theory you could represent these by two different function approximators, but this would be quite inefficient. However, note that 
<span class="math-container">$$Q(s,a) = \sum_{s',r} \mathbb{P}(s',r|s,a)(r + V(s') = \mathbb{E}[r + V(s')|a,s]\;,$$</span>
so we can actually use a single function approximation, for <span class="math-container">$V(s)$</span>, to completely represent the advantage function. To optimise this function approximator you would use the returns at each step of the episode as in e.g. the REINFORCE algorithm like you mentioned. </p>

CC BY-SA 4.0
2020-05-22T09:22:18.367Z
2020-05-22T09:22:18.367Z
David
<p>From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages <strong>increase</strong> the probabilities, negative advantages <strong>reduce</strong> the probabilities.</p>

<p>So, how do we compute the advantages given the <strong>real discounted rewards</strong> (aggregated rewards from the episode) and a <strong>policy network</strong> that only outputs the probabilities of actions?</p>

<reinforcement-learning><objective-functions><policy-gradients><reinforce>
How to calculate the advantage in policy gradient functions?
Bob Kimani
# How to calculate the advantage in policy gradient functions?

(Asked by: username_0 on 2020-03-17)

From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages **increase** the probabilities, negative advantages **reduce** the probabilities.

So, how do we compute the advantages given the **real discounted rewards** (aggregated rewards from the episode) and a **policy network** that only outputs the probabilities of actions?

**Tags:** reinforcement-learning, objective-functions, policy-gradients, reinforce


---

## 2 answers

### Answer by username_2 on 2020-05-22. Score: 2 | Accepted answer

First let us note the definition of the advantage function:

$$A(s,a) = Q(s,a) - V(s) \; ,$$

where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could represent these by two different function approximators, but this would be quite inefficient. However, note that
$$Q(s,a) = \sum\_{s',r} \mathbb{P}(s',r|s,a)(r + V(s') = \mathbb{E}[r + V(s')|a,s]\;,$$
so we can actually use a single function approximation, for $V(s)$, to completely represent the advantage function. To optimise this function approximator you would use the returns at each step of the episode as in e.g. the REINFORCE algorithm like you mentioned.


---

### Answer by username_1 on 2020-05-20. Score: 2

The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive nudge.

One way to estimate the baseline is to have a value function approximator. At every step, you train a NN, using the trajectories collected via the current policy, to predict the value function for states.

I hope that answers your query.


---

