<p>In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case</p>
<p>The network predicts its own target value, now how exactly does it converge if the network predicts its target value</p>
<p>Can someone explain this to me ??</p>
<p><a href="https://i.sstatic.net/pe3vt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pe3vt.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2020-08-07T21:33:45.603Z
Initially, the target will be totally wrong but as the process goes on and on it will become closer to  the unbiased estimate, then it will eventually converge
Swakshar Deb
CC BY-SA 4.0
2020-08-07T21:40:36.940Z
Also, remember that this target is an unbiased estimate of the optimal Q value.
Swakshar Deb
CC BY-SA 4.0
2020-08-07T22:32:01.220Z
@Swakshar Deb Why does it get closer to the unbiased estimate, what exactly makes it that way
Chukwudi Ogbonna
CC BY-SA 4.0
2020-08-07T22:32:28.593Z
@Swakshar Deb why does it get closer to the unbiased estimate, what in that formula makes it the unbiased estimate
Chukwudi Ogbonna
CC BY-SA 4.0
2020-08-08T06:25:28.717Z
$q^{*}=E[R+\gamma \max_{a}Q(s_{t+1},a_{t+1})]$, that means $R+\gamma \max_{a} Q(s_{t+1},a_{t+1})$ is the unbiased estimate of $q^{*}$. So, if the iteration is large enough the target will eventually converge to the true estimate.
Swakshar Deb
CC BY-SA 4.0
2020-08-08T07:51:06.443Z
@Swakshar Deb I would have liked the answer as a post for other users, but this is fine , Thanks a lot , I appreciate
Chukwudi Ogbonna
CC BY-SA 4.0
2020-08-08T13:07:16.043Z
@ChukwudiOgbonna Can you please describe in your own words your problem without taking screenshots? Can you edit your post to do this? If we take the time to read your posts and try to help, you should also take the time to write a question/post properly and show that you have put some effort into writing.
nbro
CC BY-SA 4.0
2020-08-08T13:07:55.840Z
@SwaksharDeb I suggest that you a formal answer to the question (if you have one: I didn't read the comments above), given that comments are meant to be temporary.
nbro
CC BY-SA 4.0
2020-08-08T13:26:02.720Z
Sorry, I don't have the mathematical proof for the convergence. I have to look for it. If I get one I'll post it.
Swakshar Deb
<reinforcement-learning><q-learning><dqn><deep-rl>
How does DQN convergence work in reinforcement learning
Chukwudi Ogbonna
# How does DQN convergence work in reinforcement learning

(Asked by: username_0 on 2020-08-07)

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case

The network predicts its own target value, now how exactly does it converge if the network predicts its target value

Can someone explain this to me ??

[![enter image description here](https://i.sstatic.net/pe3vt.png)](https://i.sstatic.net/pe3vt.png)

**Tags:** reinforcement-learning, q-learning, dqn, deep-rl

> **Comments:**
> **username_1**: Initially, the target will be totally wrong but as the process goes on and on it will become closer to  the unbiased estimate, then it will eventually converge  
> **username_1**: Also, remember that this target is an unbiased estimate of the optimal Q value.  
> **username_0**: @username_1 Why does it get closer to the unbiased estimate, what exactly makes it that way  
> **username_0**: @username_1 why does it get closer to the unbiased estimate, what in that formula makes it the unbiased estimate  
> **username_1**: $q^{*}=E[R+\gamma \max_{a}Q(s_{t+1},a_{t+1})]$, that means $R+\gamma \max_{a} Q(s_{t+1},a_{t+1})$ is the unbiased estimate of $q^{*}$. So, if the iteration is large enough the target will eventually converge to the true estimate.  
> **username_0**: @username_1 I would have liked the answer as a post for other users, but this is fine , Thanks a lot , I appreciate  
> **username_2**: @ChukwudiOgbonna Can you please describe in your own words your problem without taking screenshots? Can you edit your post to do this? If we take the time to read your posts and try to help, you should also take the time to write a question/post properly and show that you have put some effort into writing.  
> **username_2**: @SwaksharDeb I suggest that you a formal answer to the question (if you have one: I didn't read the comments above), given that comments are meant to be temporary.  
> **username_1**: Sorry, I don't have the mathematical proof for the convergence. I have to look for it. If I get one I'll post it.  

---

## 0 answers

