Body: <p>In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case</p>
  <p>The network predicts its own target value, now how exactly does it converge if the network predicts its target value</p>
  <p>Can someone explain this to me ??</p>
  <p><a href="https://i.sstatic.net/pe3vt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pe3vt.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T21:33:45.603Z
Comments.Text: Initially, the target will be totally wrong but as the process goes on and on it will become closer to  the unbiased estimate, then it will eventually converge
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T21:40:36.940Z
Comments.Text: Also, remember that this target is an unbiased estimate of the optimal Q value.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T22:32:01.220Z
Comments.Text: @Swakshar Deb Why does it get closer to the unbiased estimate, what exactly makes it that way
Comments.UserDisplayName: Chukwudi Ogbonna
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T22:32:28.593Z
Comments.Text: @Swakshar Deb why does it get closer to the unbiased estimate, what in that formula makes it the unbiased estimate
Comments.UserDisplayName: Chukwudi Ogbonna
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-08T06:25:28.717Z
Comments.Text: $q^{*}=E[R+\gamma \max_{a}Q(s_{t+1},a_{t+1})]$, that means $R+\gamma \max_{a} Q(s_{t+1},a_{t+1})$ is the unbiased estimate of $q^{*}$. So, if the iteration is large enough the target will eventually converge to the true estimate.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-08T07:51:06.443Z
Comments.Text: @Swakshar Deb I would have liked the answer as a post for other users, but this is fine , Thanks a lot , I appreciate
Comments.UserDisplayName: Chukwudi Ogbonna
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-08T13:07:16.043Z
Comments.Text: @ChukwudiOgbonna Can you please describe in your own words your problem without taking screenshots? Can you edit your post to do this? If we take the time to read your posts and try to help, you should also take the time to write a question/post properly and show that you have put some effort into writing.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-08T13:07:55.840Z
Comments.Text: @SwaksharDeb I suggest that you a formal answer to the question (if you have one: I didn't read the comments above), given that comments are meant to be temporary.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-08T13:26:02.720Z
Comments.Text: Sorry, I don't have the mathematical proof for the convergence. I have to look for it. If I get one I'll post it.
Comments.UserDisplayName: Swakshar Deb
Tags: <reinforcement-learning><q-learning><dqn><deep-rl>
Title: How does DQN convergence work in reinforcement learning
UserDisplayName: Chukwudi Ogbonna
ThreadText: # How does DQN convergence work in reinforcement learning
  
  (Asked by: username_0 on 2020-08-07)
  
  In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case
  
  The network predicts its own target value, now how exactly does it converge if the network predicts its target value
  
  Can someone explain this to me ??
  
  [![enter image description here](https://i.sstatic.net/pe3vt.png)](https://i.sstatic.net/pe3vt.png)
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl
  
  > **Comments:**
  > **username_1**: Initially, the target will be totally wrong but as the process goes on and on it will become closer to  the unbiased estimate, then it will eventually converge  
  > **username_1**: Also, remember that this target is an unbiased estimate of the optimal Q value.  
  > **username_0**: @username_1 Why does it get closer to the unbiased estimate, what exactly makes it that way  
  > **username_0**: @username_1 why does it get closer to the unbiased estimate, what in that formula makes it the unbiased estimate  
  > **username_1**: $q^{*}=E[R+\gamma \max_{a}Q(s_{t+1},a_{t+1})]$, that means $R+\gamma \max_{a} Q(s_{t+1},a_{t+1})$ is the unbiased estimate of $q^{*}$. So, if the iteration is large enough the target will eventually converge to the true estimate.  
  > **username_0**: @username_1 I would have liked the answer as a post for other users, but this is fine , Thanks a lot , I appreciate  
  > **username_2**: @username_0 Can you please describe in your own words your problem without taking screenshots? Can you edit your post to do this? If we take the time to read your posts and try to help, you should also take the time to write a question/post properly and show that you have put some effort into writing.  
  > **username_2**: @username_1 I suggest that you a formal answer to the question (if you have one: I didn't read the comments above), given that comments are meant to be temporary.  
  > **username_1**: Sorry, I don't have the mathematical proof for the convergence. I have to look for it. If I get one I'll post it.  
  
  ---
  
  ## 0 answers
  
  