Answers.Body: <p>The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive nudge.</p>
  
  <p>One way to estimate the baseline is to have a value function approximator. At every step, you train a NN, using the trajectories collected via the current policy, to predict the value function for states.  </p>
  
  <p>I hope that answers your query.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-20T22:03:53.197Z
Answers.LastActivityDate: 2020-05-21T17:22:03.080Z
Answers.LastEditDate: 2020-05-21T17:22:03.080Z
Answers.UserDisplayName: pecey
Answers.Body: <p>First let us note the definition of the advantage function: </p>
  
  <p><span class="math-container">$$A(s,a) = Q(s,a) - V(s) \; ,$$</span></p>
  
  <p>where <span class="math-container">$Q(s,a)$</span> is the action-value function and <span class="math-container">$V(s)$</span> is the state-value function. In theory you could represent these by two different function approximators, but this would be quite inefficient. However, note that 
  <span class="math-container">$$Q(s,a) = \sum_{s',r} \mathbb{P}(s',r|s,a)(r + V(s') = \mathbb{E}[r + V(s')|a,s]\;,$$</span>
  so we can actually use a single function approximation, for <span class="math-container">$V(s)$</span>, to completely represent the advantage function. To optimise this function approximator you would use the returns at each step of the episode as in e.g. the REINFORCE algorithm like you mentioned. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-22T09:22:18.367Z
Answers.LastActivityDate: 2020-05-22T09:22:18.367Z
Answers.UserDisplayName: David
Body: <p>From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages <strong>increase</strong> the probabilities, negative advantages <strong>reduce</strong> the probabilities.</p>
  
  <p>So, how do we compute the advantages given the <strong>real discounted rewards</strong> (aggregated rewards from the episode) and a <strong>policy network</strong> that only outputs the probabilities of actions?</p>
  
Tags: <reinforcement-learning><objective-functions><policy-gradients><reinforce>
Title: How to calculate the advantage in policy gradient functions?
UserDisplayName: Bob Kimani
ThreadText: # How to calculate the advantage in policy gradient functions?
  
  (Asked by: username_0 on 2020-03-17)
  
  From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages **increase** the probabilities, negative advantages **reduce** the probabilities.
  
  So, how do we compute the advantages given the **real discounted rewards** (aggregated rewards from the episode) and a **policy network** that only outputs the probabilities of actions?
  
  **Tags:** reinforcement-learning, objective-functions, policy-gradients, reinforce
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-05-22. Score: 2 | Accepted answer
  
  First let us note the definition of the advantage function:
  
  $$A(s,a) = Q(s,a) - V(s) \; ,$$
  
  where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could represent these by two different function approximators, but this would be quite inefficient. However, note that
  $$Q(s,a) = \sum\_{s',r} \mathbb{P}(s',r|s,a)(r + V(s') = \mathbb{E}[r + V(s')|a,s]\;,$$
  so we can actually use a single function approximation, for $V(s)$, to completely represent the advantage function. To optimise this function approximator you would use the returns at each step of the episode as in e.g. the REINFORCE algorithm like you mentioned.
  
  
  ---
  
  ### Answer by username_1 on 2020-05-20. Score: 2
  
  The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive nudge.
  
  One way to estimate the baseline is to have a value function approximator. At every step, you train a NN, using the trajectories collected via the current policy, to predict the value function for states.
  
  I hope that answers your query.
  
  
  ---
  
  