Answers.Body: <p>Let's say <span class="math-container">$Q$</span> is the old estimate, <span class="math-container">$Q'$</span> the new estimate, and <span class="math-container">$R$</span> is the return.</p>
  <p>We have</p>
  <p><span class="math-container">$$ 
  Q' = Q + \alpha(R-Q)
  $$</span></p>
  <p>This can be rewritten as</p>
  <p><span class="math-container">$$ 
  Q' = (1-\alpha)Q + \alpha R
  $$</span></p>
  <p>When <span class="math-container">$\alpha$</span> is a constant, this is an exponential weighted average of returns. If <span class="math-container">$n$</span> is the number of samples we get and <span class="math-container">$\alpha=1/n$</span> (so it decreases with each sample), we get</p>
  <p><span class="math-container">$$
  Q' = \frac{n-1}{n}Q + \frac{1}{n}R
  $$</span></p>
  <p>This simply represents the average return. So, playing with <span class="math-container">$\alpha$</span> tunes the weighting of the estimate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-22T13:02:58.633Z
Answers.Text: You’re completely missing me, I understand the weighted average in TD learning for normal Q learning, my question is how is this computed with deep Q networks ???
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T01:04:21.297Z
Answers.Text: Firstly, please don't leave me comments to check your comments.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T01:10:14.470Z
Answers.Text: In deep Q learning, you do function approximation. You make estimates of the target returns and current Q values from samples (which is an unbiased estimator of the mean). Then you simply use SGD to minimize the MSE between your current state estimate and your target. As you mentioned, this is in fact an approximation. In general there are no convergence guarantees in deep RL. However, in the limit of infinite data, you can suppose your target network will converge to the value function (assuming the network is wide enough).
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T02:10:45.653Z
Answers.Text: I still don’t get it,I know the Q values are unbiased estimates of the optimal values, the expectation which is the mean is the weighted average, which is the update formula in normal Q learning, in Deep Q learning, how is that weighted average implemented?,that’s my question , do you understand ?
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T02:25:40.503Z
Answers.Text: What I mean exactly is, in Q learning the update rule implements the Weighted average which is the expectation,  in deep q learning how exactly is the weighted average implemented
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T02:36:39.827Z
Answers.Text: I see. I believe the answer to this is that it isn't in fact implemented in deep Q learning. In the weighted average equation you mentioned, it's simple to implement with tabular methods because you update the Q values state/action-wise. With uncountable state space, you can't really do this. Instead you make updates in parameter space, so you're never directly updating particular state/action values.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T02:42:06.367Z
Answers.Text: Wow, so if that’s the case, how are the expected values computed in Deep Q learning , because expectation is basically the mean right, so in Deep Q learning, we’re using a target which is the unbiased estimate of the true target Value, so how is the expectation computed in DQNs
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T02:44:01.693Z
Answers.Text: What I mean exactly is the target we use is the unbiased estimate of the actual target, meaning the expectation of the target gives the true target value, so my question is in DQNs how is that expectation computed, in normal Q leaning it’s the weighted average, in DQNs how is that computed
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T13:49:35.740Z
Answers.Text: In DQN it's computed so as to minimize the expected MSE.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-22T12:58:57.710Z
Answers.LastActivityDate: 2020-08-22T12:58:57.710Z
Answers.UserDisplayName: harwiltz
Body: <p>I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns
  I.e</p>
  <p>New estimate= old estimate +alpha*[returns- old estimate]</p>
  <p>So by the law of large numbers this will converge to the optimal true q value</p>
  <p>Now when we go to Deep Q networks,how exactly is the weighted average computed, all they simply did was try to reduce the error between the target and the estimate, and keep in mind this isn’t the true target, it’s just an unbiased estimate,since it’s an unbiased estimate how is the weighted average computed , which is the expectation?</p>
  <p>Can someone help me out here??
  Thanks in advance</p>
  
Tags: <reinforcement-learning><deep-learning><q-learning><dqn><deep-neural-networks>
Title: How is weighted average computed in Deep Q networks
UserDisplayName: Chukwudi Ogbonna
ThreadText: # How is weighted average computed in Deep Q networks
  
  (Asked by: username_0 on 2020-08-22)
  
  I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns
  I.e
  
  New estimate= old estimate +alpha\*[returns- old estimate]
  
  So by the law of large numbers this will converge to the optimal true q value
  
  Now when we go to Deep Q networks,how exactly is the weighted average computed, all they simply did was try to reduce the error between the target and the estimate, and keep in mind this isn’t the true target, it’s just an unbiased estimate,since it’s an unbiased estimate how is the weighted average computed , which is the expectation?
  
  Can someone help me out here??
  Thanks in advance
  
  **Tags:** reinforcement-learning, deep-learning, q-learning, dqn, deep-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-22. Score: 2
  
  Let's say $Q$ is the old estimate, $Q'$ the new estimate, and $R$ is the return.
  
  We have
  
  $$
  Q' = Q + \alpha(R-Q)
  $$
  
  This can be rewritten as
  
  $$
  Q' = (1-\alpha)Q + \alpha R
  $$
  
  When $\alpha$ is a constant, this is an exponential weighted average of returns. If $n$ is the number of samples we get and $\alpha=1/n$ (so it decreases with each sample), we get
  
  $$
  Q' = \frac{n-1}{n}Q + \frac{1}{n}R
  $$
  
  This simply represents the average return. So, playing with $\alpha$ tunes the weighting of the estimate.
  
  > **Comments:**
  > **username_0**: You’re completely missing me, I understand the weighted average in TD learning for normal Q learning, my question is how is this computed with deep Q networks ???  
  > **username_1**: Firstly, please don't leave me comments to check your comments.  
  > **username_1**: In deep Q learning, you do function approximation. You make estimates of the target returns and current Q values from samples (which is an unbiased estimator of the mean). Then you simply use SGD to minimize the MSE between your current state estimate and your target. As you mentioned, this is in fact an approximation. In general there are no convergence guarantees in deep RL. However, in the limit of infinite data, you can suppose your target network will converge to the value function (assuming the network is wide enough).  
  > **username_0**: I still don’t get it,I know the Q values are unbiased estimates of the optimal values, the expectation which is the mean is the weighted average, which is the update formula in normal Q learning, in Deep Q learning, how is that weighted average implemented?,that’s my question , do you understand ?  
  > **username_0**: What I mean exactly is, in Q learning the update rule implements the Weighted average which is the expectation,  in deep q learning how exactly is the weighted average implemented  
  > **username_1**: I see. I believe the answer to this is that it isn't in fact implemented in deep Q learning. In the weighted average equation you mentioned, it's simple to implement with tabular methods because you update the Q values state/action-wise. With uncountable state space, you can't really do this. Instead you make updates in parameter space, so you're never directly updating particular state/action values.  
  > **username_0**: Wow, so if that’s the case, how are the expected values computed in Deep Q learning , because expectation is basically the mean right, so in Deep Q learning, we’re using a target which is the unbiased estimate of the true target Value, so how is the expectation computed in DQNs  
  > **username_0**: What I mean exactly is the target we use is the unbiased estimate of the actual target, meaning the expectation of the target gives the true target value, so my question is in DQNs how is that expectation computed, in normal Q leaning it’s the weighted average, in DQNs how is that computed  
  > **username_1**: In DQN it's computed so as to minimize the expected MSE.  
  
  ---
  
  