Answers.Body: <p>I can't give a conclusive answer, but here's an attempt, largely based on <a href="https://arxiv.org/abs/1503.04269" rel="nofollow noreferrer">An Empathic Approach to the Problem of Off-policy Temporal-Difference Learning</a>. With function approximation, updating <span class="math-container">$V_\pi$</span> for state <span class="math-container">$s_t$</span> due to a recorded transition may update state <span class="math-container">$s^\prime$</span>. Lets say the value estimate for <span class="math-container">$s_t$</span> is 1 and the estimate for <span class="math-container">$s^\prime$</span> is 2 and we have a linear network, with <span class="math-container">$\gamma \approx 1$</span> and learning rate <span class="math-container">$0.1$</span>.
  Lets also say we have recorded a transition that gives 0 reward.</p>
  <p>In the off-policy setting, when learning from this transition, first we update the value for <span class="math-container">$s_t$</span> to 1.1 (and <span class="math-container">$s^\prime$</span> to 2.2). The second time the transition is used we update to 1.21 (and 2.42). Repeated off-policy learning can basically cause the learning to diverge as updates get larger. A target network fixes the target value for a longer period, which may mitigate this - the target remains at 2 for a longer time which reduces the updates - but it's still vulnerable to this issue.</p>
  <p>In the <em>on-policy</em> setting, once we reach <span class="math-container">$s^\prime$</span> we need to learn from a new transition that starts in <span class="math-container">$s^\prime$</span>, and our current overestimation bias of the value of <span class="math-container">$s^\prime$</span> will impact the new error, directly, hopefully counteracting the bias. This is why there are much stronger convergence guarantees in the on policy case for function approximation and bootstrapping. There is some research into target functions for on policy actor-critic but I don't think it has been shown to be more effective, rather the opposite.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-12T19:50:04.803Z
Answers.LastActivityDate: 2022-11-12T19:50:04.803Z
Answers.UserDisplayName: Dennis Malmgren
Body: <p>Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function <span class="math-container">$V$</span>. The Bellman equation for the value function is as bellow:</p>
  <p><span class="math-container">$$
  V_\pi(s_t) = \sum_a \pi(a|s_t)\sum_{r, s'}(r+V_\pi(s'))P(s', r|s, a)
  $$</span></p>
  <p>It makes sense not to have a replay buffer due to the current policy in the formula and the fact that our approach is on-policy. However, I really do not figure out why no one uses a target network to stabilize the training process of the critic, like what we have in DQN, namely the variant published in 2015. Does anyone have an idea for that with probably a citation?</p>
  <p>I know that DDPG uses a critic with a fixed target network, but be aware that it is a real off-policy actor-critic. By &quot;real&quot; I mean it is not due to importance sampling.</p>
  <p>I have to mention that I can imagine something, but I'm not sure whether it is true or not. If we have a target network, it means we are trying to find a deterministic, optimal in the case of DQN, policy, while we are learning the current policy's data for the actor-critic case with the critic.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-24T21:55:51.953Z
Comments.Text: Common boys! Doesn't anyone want the bounty?! The question is not strange. It is something that everyone considers in the papers.
Comments.UserDisplayName: Green Falcon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-25T01:31:21.670Z
Comments.Text: Just to contextualize more, what is an example of an on-policy AC method that does not use the target network that you are referring to?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-25T07:29:12.353Z
Comments.Text: @nbro Vanilla on-policy AC does not use target for the critic. Available in textbooks and source codes.
Comments.UserDisplayName: Green Falcon
Tags: <reinforcement-learning><deep-rl><dqn><actor-critic-methods><target-network>
Title: Why isn't a target network used for the critic in on-policy actor-critic methods?
UserDisplayName: Green Falcon
ThreadText: # Why isn't a target network used for the critic in on-policy actor-critic methods?
  
  (Asked by: username_0 on 2021-11-07)
  
  Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow:
  
  $$
  V\_\pi(s\_t) = \sum\_a \pi(a|s\_t)\sum\_{r, s'}(r+V\_\pi(s'))P(s', r|s, a)
  $$
  
  It makes sense not to have a replay buffer due to the current policy in the formula and the fact that our approach is on-policy. However, I really do not figure out why no one uses a target network to stabilize the training process of the critic, like what we have in DQN, namely the variant published in 2015. Does anyone have an idea for that with probably a citation?
  
  I know that DDPG uses a critic with a fixed target network, but be aware that it is a real off-policy actor-critic. By "real" I mean it is not due to importance sampling.
  
  I have to mention that I can imagine something, but I'm not sure whether it is true or not. If we have a target network, it means we are trying to find a deterministic, optimal in the case of DQN, policy, while we are learning the current policy's data for the actor-critic case with the critic.
  
  **Tags:** reinforcement-learning, deep-rl, dqn, actor-critic-methods, target-network
  
  > **Comments:**
  > **username_0**: Common boys! Doesn't anyone want the bounty?! The question is not strange. It is something that everyone considers in the papers.  
  > **username_1**: Just to contextualize more, what is an example of an on-policy AC method that does not use the target network that you are referring to?  
  > **username_0**: @username_1 Vanilla on-policy AC does not use target for the critic. Available in textbooks and source codes.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-12. Score: 1
  
  I can't give a conclusive answer, but here's an attempt, largely based on [An Empathic Approach to the Problem of Off-policy Temporal-Difference Learning](https://arxiv.org/abs/1503.04269). With function approximation, updating $V\_\pi$ for state $s\_t$ due to a recorded transition may update state $s^\prime$. Lets say the value estimate for $s\_t$ is 1 and the estimate for $s^\prime$ is 2 and we have a linear network, with $\gamma \approx 1$ and learning rate $0.1$.
  Lets also say we have recorded a transition that gives 0 reward.
  
  In the off-policy setting, when learning from this transition, first we update the value for $s\_t$ to 1.1 (and $s^\prime$ to 2.2). The second time the transition is used we update to 1.21 (and 2.42). Repeated off-policy learning can basically cause the learning to diverge as updates get larger. A target network fixes the target value for a longer period, which may mitigate this - the target remains at 2 for a longer time which reduces the updates - but it's still vulnerable to this issue.
  
  In the *on-policy* setting, once we reach $s^\prime$ we need to learn from a new transition that starts in $s^\prime$, and our current overestimation bias of the value of $s^\prime$ will impact the new error, directly, hopefully counteracting the bias. This is why there are much stronger convergence guarantees in the on policy case for function approximation and bootstrapping. There is some research into target functions for on policy actor-critic but I don't think it has been shown to be more effective, rather the opposite.
  
  
  ---
  
  