Answers.Body: <p>Especially in continuous space, convergence of the value function is mainly a theoretical property. Without seeing enough of the state space, as you suggest, there's no way to ensure that your Q function will generalize to the whole state space. Convergence results for Q learning with function approximation generally show that in the limit of infinite data, your value function will converge to the desired fixed point -- note that this is only true when your agent explores occasionally, for an infinite amount of time.</p>
  <p>When your parameters have converged, this simply means that your Q function has fit the data you've collected. As you explore more, your agent may get &quot;surprised&quot; and your parameters may start to change again.</p>
  <p>Also, convergence of the parameters in function approximation can never guarantee that an optimal value function was found in practice -- the only guarantee you can wish for is that the optimal value function <em>that can be produced with your model</em> has been found. For instance, the parameters of the linear Q function you posted can converge, even if the optimal Q function is not linear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T00:46:15.567Z
Answers.LastActivityDate: 2020-06-21T00:46:15.567Z
Answers.UserDisplayName: harwiltz
Answers.Body: <p>Well, you want your network to have a good prediction powers for the Q-values. So you compare Q-value at time t with the reward that you've got at time t after having executed action a + the prediction of the best Q-value of your neural network at time t+1. Note, that you are optimizing using a prediction and not a true value. That is called bootstrapping, look up TD-learning to have a better grasp of the concept.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-22T20:37:44.047Z
Answers.LastActivityDate: 2020-01-22T20:37:44.047Z
Answers.UserDisplayName: d56
Body: <p>I am trying to understand how deep Q learning (DQN) works. To my current understanding, each <span class="math-container">$Q(s, a)$</span> functions is estimated to be a function of a feature vector of its state <span class="math-container">$\phi$</span>(s) and the weight of the network <span class="math-container">$\theta$</span>.</p>
  <p>The loss function to minimise is <span class="math-container">$||\delta_{t+1}||^2$</span> where <span class="math-container">$\delta_{t+1}$</span> is shown below. The loss function is from the website talking about function approximation. Even though it is not explicitly deep Q learning, the loss function to minimise is similar.</p>
  <p><span class="math-container">$$\delta_{\mathrm{t}+1}=\mathrm{R}_{\mathrm{t}+1}+\max _{\mathrm{a}\in\mathrm{A}} \boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{t+1}, \mathrm{a}\right)-\boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{\mathrm{t}}, \mathrm{a}\right)$$</span></p>
  <p>Source: <a href="https://towardsdatascience.com/function-approximation-in-reinforcement-learning-85a4864d566" rel="nofollow noreferrer">https://towardsdatascience.com/function-approximation-in-reinforcement-learning-85a4864d566</a>.</p>
  <p>Intuitively, I am not able to understand why the loss function is defined as such. Once the network converges to a <span class="math-container">$\theta$</span> using gradient descent, does that mean that the <span class="math-container">$Q_{max}(s,a)$</span> is found?</p>
  <p>In essence, I am not able to grasp intuitively how the neural network is able to generalise the learning to unseen states.</p>
  <p>The algorithm I am looking at to help me understand the deep Q networks is below.</p>
  <p><a href="https://i.sstatic.net/DSsAU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DSsAU.png" alt="enter image description here" /></a></p>
  <p>Source: <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-14T14:41:21.860Z
Comments.Text: I've tried to rewrite the title so that it contains what I think is the actual question. Make sure that the title contains what was your actual question.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><dqn><objective-functions><generalization>
Title: How is the DQN able to generalise the learning to unseen states with such a loss function?
UserDisplayName: calveeen
ThreadText: # How is the DQN able to generalise the learning to unseen states with such a loss function?
  
  (Asked by: username_0 on 2020-01-22)
  
  I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight of the network $\theta$.
  
  The loss function to minimise is $||\delta\_{t+1}||^2$ where $\delta\_{t+1}$ is shown below. The loss function is from the website talking about function approximation. Even though it is not explicitly deep Q learning, the loss function to minimise is similar.
  
  $$\delta\_{\mathrm{t}+1}=\mathrm{R}\_{\mathrm{t}+1}+\max \_{\mathrm{a}\in\mathrm{A}} \boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}\_{t+1}, \mathrm{a}\right)-\boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}\_{\mathrm{t}}, \mathrm{a}\right)$$
  
  Source: <https://towardsdatascience.com/function-approximation-in-reinforcement-learning-85a4864d566>.
  
  Intuitively, I am not able to understand why the loss function is defined as such. Once the network converges to a $\theta$ using gradient descent, does that mean that the $Q\_{max}(s,a)$ is found?
  
  In essence, I am not able to grasp intuitively how the neural network is able to generalise the learning to unseen states.
  
  The algorithm I am looking at to help me understand the deep Q networks is below.
  
  [![enter image description here](https://i.sstatic.net/DSsAU.png)](https://i.sstatic.net/DSsAU.png)
  
  Source: [https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf)
  
  **Tags:** reinforcement-learning, deep-rl, dqn, objective-functions, generalization
  > **Comments:**
  > **username_1**: I've tried to rewrite the title so that it contains what I think is the actual question. Make sure that the title contains what was your actual question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-06-21. Score: 1
  
  Especially in continuous space, convergence of the value function is mainly a theoretical property. Without seeing enough of the state space, as you suggest, there's no way to ensure that your Q function will generalize to the whole state space. Convergence results for Q learning with function approximation generally show that in the limit of infinite data, your value function will converge to the desired fixed point -- note that this is only true when your agent explores occasionally, for an infinite amount of time.
  
  When your parameters have converged, this simply means that your Q function has fit the data you've collected. As you explore more, your agent may get "surprised" and your parameters may start to change again.
  
  Also, convergence of the parameters in function approximation can never guarantee that an optimal value function was found in practice -- the only guarantee you can wish for is that the optimal value function *that can be produced with your model* has been found. For instance, the parameters of the linear Q function you posted can converge, even if the optimal Q function is not linear.
  
  
  ---
  
  ### Answer by username_3 on 2020-01-22. Score: 0
  
  Well, you want your network to have a good prediction powers for the Q-values. So you compare Q-value at time t with the reward that you've got at time t after having executed action a + the prediction of the best Q-value of your neural network at time t+1. Note, that you are optimizing using a prediction and not a true value. That is called bootstrapping, look up TD-learning to have a better grasp of the concept.
  
  
  ---
  
  