Answers.Body: <p>The policy doesn't change over time. That is, the values will change, otherwise we would not be learning anything, but our rules for action selection don't. I.e. we always take action according to the distribution postulated to our current estimate of the policy <span class="math-container">$\pi_\theta(a|s)$</span>, we don't suddenly start taking <span class="math-container">$\max_a \pi_\theta(a|s)$</span>, which would be a true change in policy and would make learning both the actor and the critic unstable.</p>
  <p>Because NN's are able to handle noisy target distributions, this is how they can deal with the changing data. If you think of how Actor-Critic methods work, you would initially start to shift your NN to some unfeasible values (due to random initialisation and the Actor-Critic not having any information about the environment), but as you start to interact with the environment you will start to update the agent towards the 'true' policy.</p>
  <p>An analogy in supervised learning would be to have some noisy data which is incorrect and some true data. If you trained your network on the noisy data for a small number of epochs and then never showed it to the network again and trained it solely on the correct data, it would forget it has ever seen the noisy data and only represent the new, true data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T09:38:18.830Z
Answers.Text: The rules for *derivation* of the target policy do not change. However, the target policy **does** change over time in both cases (Q learning and actor-critic) as estimates are refined. In turn that means the value functions being learned also change over time in both cases the OP is interested in. The reason it works is that NNs can deal with non-stationary target distributions, forgetting the old values.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T09:40:08.417Z
Answers.Text: @NeilSlater I meant more that the policy doesn't change in the sense that we take actions according to the distribution (which will change) consistently, we don't suddenly start taking e.g. $\max_a \pi_{\theta}(a|s)$.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T09:41:02.127Z
Answers.Text: Yes, I was justclarifying that in my comment. OP seems to have conflated derivation rule not changing and policy not changing though, so it is important to be clear IMO
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T09:41:42.107Z
Answers.Text: right -- I will edit my answer to reflect this.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T09:36:02.220Z
Answers.LastActivityDate: 2020-07-27T09:47:39.823Z
Answers.LastEditDate: 2020-07-27T09:47:39.823Z
Answers.UserDisplayName: David
Body: <p>According to deep Q learning, we want to learn <span class="math-container">$Q^*(s,a)$</span>, which is the optimal action-value function. It does make sense because we assume there is only one optimal function so the algorithm will converge supposedly.</p>
  <p>But when it comes to actor-critic method, we use critic network (also called  value network) to estimate <span class="math-container">$Q_\pi(s, a)$</span>. This is what confused me. Since our policy <span class="math-container">$\pi$</span> will change through time, the target <span class="math-container">$Q_\pi(s, a)$</span> of value network will also change. What will happen for a network to learn a changing function?</p>
  
Tags: <machine-learning><reinforcement-learning><deep-learning>
Title: Why can we use a network to estimate $Q_\pi(s, a)$ in Actor-Critic Method?
UserDisplayName: KningTG
ThreadText: # Why can we use a network to estimate $Q_\pi(s, a)$ in Actor-Critic Method?
  
  (Asked by: username_0 on 2020-07-27)
  
  According to deep Q learning, we want to learn $Q^\*(s,a)$, which is the optimal action-value function. It does make sense because we assume there is only one optimal function so the algorithm will converge supposedly.
  
  But when it comes to actor-critic method, we use critic network (also called value network) to estimate $Q\_\pi(s, a)$. This is what confused me. Since our policy $\pi$ will change through time, the target $Q\_\pi(s, a)$ of value network will also change. What will happen for a network to learn a changing function?
  
  **Tags:** machine-learning, reinforcement-learning, deep-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-27. Score: 1
  
  The policy doesn't change over time. That is, the values will change, otherwise we would not be learning anything, but our rules for action selection don't. I.e. we always take action according to the distribution postulated to our current estimate of the policy $\pi\_\theta(a|s)$, we don't suddenly start taking $\max\_a \pi\_\theta(a|s)$, which would be a true change in policy and would make learning both the actor and the critic unstable.
  
  Because NN's are able to handle noisy target distributions, this is how they can deal with the changing data. If you think of how Actor-Critic methods work, you would initially start to shift your NN to some unfeasible values (due to random initialisation and the Actor-Critic not having any information about the environment), but as you start to interact with the environment you will start to update the agent towards the 'true' policy.
  
  An analogy in supervised learning would be to have some noisy data which is incorrect and some true data. If you trained your network on the noisy data for a small number of epochs and then never showed it to the network again and trained it solely on the correct data, it would forget it has ever seen the noisy data and only represent the new, true data.
  
  > **Comments:**
  > **username_2**: The rules for *derivation* of the target policy do not change. However, the target policy **does** change over time in both cases (Q learning and actor-critic) as estimates are refined. In turn that means the value functions being learned also change over time in both cases the OP is interested in. The reason it works is that NNs can deal with non-stationary target distributions, forgetting the old values.  
  > **username_1**: @NeilSlater I meant more that the policy doesn't change in the sense that we take actions according to the distribution (which will change) consistently, we don't suddenly start taking e.g. $\max_a \pi_{\theta}(a|s)$.  
  > **username_2**: Yes, I was justclarifying that in my comment. OP seems to have conflated derivation rule not changing and policy not changing though, so it is important to be clear IMO  
  > **username_1**: right -- I will edit my answer to reflect this.  
  
  ---
  
  