Answers.Body: <p>So first, you are absolutely right that both are possible but using the advantage reduces the variance and therefore speeds up the learning. I'm going to explain this with the REINFORCE algorithm.</p>
  <p>The Policy Gradient theorem provides us with a simple formula of the gradient of a parameterized  state-value function
  <span class="math-container">\begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[\sum_{a} q_{\pi}(S_{t}, a) \nabla_{\boldsymbol{\theta}} \pi(a|S_{t}, \boldsymbol{\theta}) \right]
  \end{equation}</span>
  Where <span class="math-container">$\boldsymbol{\theta}$</span> is the parameter vector, <span class="math-container">$\pi(a | s, \boldsymbol{\theta})$</span> is the probability of the policy <span class="math-container">$\pi$</span> to choose action <span class="math-container">$a$</span> in the state <span class="math-container">$s$</span> while the policy is parameterized by <span class="math-container">$\boldsymbol{\theta}$</span>.
  <br>
  The REINFORCE algorithm uses this proportionality and simplifies it even more, to
  <span class="math-container">\begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[G_{t} \nabla_{\boldsymbol{\theta}}\text{ln}(\pi(A_{t}|S_{t},\boldsymbol{\theta}))  \right]
  \end{equation}</span>
  which is translates exactly (just remove the <span class="math-container">$\nabla$</span>-operator from the right and the left hand-side) to the loss function you mentioned what you mentioned:<br>
  <code>loss = -1 * policy.logprob(action) * value</code></p>
  <p>The REINFORCE algorithm can be generalized by adding a baseline which leads to: <span class="math-container">\begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[(G_{t} - b(s)) \nabla_{\boldsymbol{\theta}}\text{ln}(\pi(A_{t}|S_{t},\boldsymbol{\theta}))  \right]
  \end{equation}</span>
  with a state-dependent baseline <span class="math-container">$b(s)$</span>. Note that this baseline cannot be dependent on the state, otherwise the Policy Gradient theorem does not hold. Since the future rewards <span class="math-container">$G_{t}$</span> vary with the state, it showed that varying the baseline with the state led to better results, therefore a natural choice is the state value function <span class="math-container">$v(S_{t})$</span>. The reason why this reduces the variance is, that the <span class="math-container">$G_{t}$</span> for REINFORCE and <span class="math-container">$G_{t} - b(s)$</span> for REINFORCE with baseline just changes the prefactor of the gradient. High gradients (removing the baseline) also leads to greedier learning but can also be more susceptible to errors. Think of it as the analogue to a (seperate) learning-rate (for each state) in classical deep learning.</p>
  <p>Coming back to q-learning vs. advantage learning we can see this as applying the Policy Gradient theorem vs. applying the policy gradient theorem with the state value function as a baseline.</p>
  <p><span class="math-container">\begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[\sum_{a} (q_{\pi}(S_{t}, a) \color{green}{ - v_{\pi}(S_{t})}) \nabla_{\boldsymbol{\theta}} \pi(a|S_{t}, \boldsymbol{\theta}) \right]
  \end{equation}</span></p>
  <p>As we know the prefactor to the gradient is the advantage:
  <span class="math-container">\begin{equation}
  A(s, a) = q(s, a) - v(s)
  \end{equation}</span></p>
  <p>TLDR:
  Advantage learning is basically q-learning with a baseline, which reduces the variance and therefore speeds up learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-06T18:10:54.107Z
Answers.LastActivityDate: 2023-02-06T18:10:54.107Z
Answers.UserDisplayName: pythonic833
Body: <p>Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps.</p>
  <p>For instance, the actor loss for a simple actor critic algorithm is given by -</p>
  <p><code>loss = -1 * policy.logprob(action) * advantage</code>,</p>
  <p>where advantage = returns - value (here <code>value</code> is estimated using a function approximator which is the critic).</p>
  <p>I don't see why the loss function just couldn't be -</p>
  <p><code>loss = -1 * policy.logprob(action) * value</code></p>
  <p><a href="http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/advantage.html" rel="nofollow noreferrer">This</a> article states the following</p>
  <blockquote>
  <p>Advantage learning and Q-learning learn equally quickly when used with a lookup table. Advantage learning can learn many orders of magnitude faster than Q-learning in some cases where a function approximator is used, even a linear function approximator. Specifically, if time steps are &quot;small&quot; in the sense that the state changes a very small amount on each time step, then advantage learning would be expected to learn much faster than Q-learning.</p>
  </blockquote>
  <p>But I don't know <em>why</em> advantage learning is faster than q-learning.</p>
  
Tags: <reinforcement-learning><deep-rl><q-learning><advantage-actor-critic>
Title: Why does Advantage Learning help function approximators?
UserDisplayName: desert_ranger
ThreadText: # Why does Advantage Learning help function approximators?
  
  (Asked by: username_0 on 2023-02-03)
  
  Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps.
  
  For instance, the actor loss for a simple actor critic algorithm is given by -
  
  `loss = -1 * policy.logprob(action) * advantage`,
  
  where advantage = returns - value (here `value` is estimated using a function approximator which is the critic).
  
  I don't see why the loss function just couldn't be -
  
  `loss = -1 * policy.logprob(action) * value`
  
  [This](http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/advantage.html) article states the following
  
  > Advantage learning and Q-learning learn equally quickly when used with a lookup table. Advantage learning can learn many orders of magnitude faster than Q-learning in some cases where a function approximator is used, even a linear function approximator. Specifically, if time steps are "small" in the sense that the state changes a very small amount on each time step, then advantage learning would be expected to learn much faster than Q-learning.
  
  But I don't know *why* advantage learning is faster than q-learning.
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, advantage-actor-critic
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-06. Score: 2 | Accepted answer
  
  So first, you are absolutely right that both are possible but using the advantage reduces the variance and therefore speeds up the learning. I'm going to explain this with the REINFORCE algorithm.
  
  The Policy Gradient theorem provides us with a simple formula of the gradient of a parameterized state-value function
  \begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}\_{\pi}\left[\sum\_{a} q\_{\pi}(S\_{t}, a) \nabla\_{\boldsymbol{\theta}} \pi(a|S\_{t}, \boldsymbol{\theta}) \right]
  \end{equation}
  Where $\boldsymbol{\theta}$ is the parameter vector, $\pi(a | s, \boldsymbol{\theta})$ is the probability of the policy $\pi$ to choose action $a$ in the state $s$ while the policy is parameterized by $\boldsymbol{\theta}$.
    
  The REINFORCE algorithm uses this proportionality and simplifies it even more, to
  \begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}\_{\pi}\left[G\_{t} \nabla\_{\boldsymbol{\theta}}\text{ln}(\pi(A\_{t}|S\_{t},\boldsymbol{\theta})) \right]
  \end{equation}
  which is translates exactly (just remove the $\nabla$-operator from the right and the left hand-side) to the loss function you mentioned what you mentioned:  
  `loss = -1 * policy.logprob(action) * value`
  
  The REINFORCE algorithm can be generalized by adding a baseline which leads to: \begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}\_{\pi}\left[(G\_{t} - b(s)) \nabla\_{\boldsymbol{\theta}}\text{ln}(\pi(A\_{t}|S\_{t},\boldsymbol{\theta})) \right]
  \end{equation}
  with a state-dependent baseline $b(s)$. Note that this baseline cannot be dependent on the state, otherwise the Policy Gradient theorem does not hold. Since the future rewards $G\_{t}$ vary with the state, it showed that varying the baseline with the state led to better results, therefore a natural choice is the state value function $v(S\_{t})$. The reason why this reduces the variance is, that the $G\_{t}$ for REINFORCE and $G\_{t} - b(s)$ for REINFORCE with baseline just changes the prefactor of the gradient. High gradients (removing the baseline) also leads to greedier learning but can also be more susceptible to errors. Think of it as the analogue to a (seperate) learning-rate (for each state) in classical deep learning.
  
  Coming back to q-learning vs. advantage learning we can see this as applying the Policy Gradient theorem vs. applying the policy gradient theorem with the state value function as a baseline.
  
  \begin{equation}
  \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}\_{\pi}\left[\sum\_{a} (q\_{\pi}(S\_{t}, a) \color{green}{ - v\_{\pi}(S\_{t})}) \nabla\_{\boldsymbol{\theta}} \pi(a|S\_{t}, \boldsymbol{\theta}) \right]
  \end{equation}
  
  As we know the prefactor to the gradient is the advantage:
  \begin{equation}
  A(s, a) = q(s, a) - v(s)
  \end{equation}
  
  TLDR:
  Advantage learning is basically q-learning with a baseline, which reduces the variance and therefore speeds up learning.
  
  
  ---
  
  