Body: <p>Hello :) I'm required to write a document where I describe what DQL does in short.
  This is what I wrote:</p>
  <p>DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a given state. Bear in mind that, we seek to find the optimal Q-value for a state-action pair at a time slot t, written as  <span class="math-container">$Q^* (s_t,a_t)$</span>, which obeys the Bellman equation:
  <span class="math-container">$$Q^*(s_t, a_t)=E\left [ R_{a_t}(s_t,s')+\gamma max_{a_{t+1}}Q^*(s_{t+1},a_{t+1}) \right ]$$</span></p>
  <p>Where <span class="math-container">$Q^* (s_{t+1},a_{t+1})$</span> is the optimal Q-value for the next state-action pair <span class="math-container">$(s_{t+1},a_{t+1})$</span>. The parameters of the DNN are updated every time step t, aiming to find the optimal policy <span class="math-container">$π^*=argmax_{a_t} Q^* (s_t,a_t)$</span> that maximizes the reward.</p>
  <p>Is what I wrote correct? Should I say <code>we seek to find the optimal Q-value for a state action pair at a time slot t</code> or should I say <code>we seek to train the DNN to resemble the optimal Q-value for a state action pair at a time slot t</code>? Which one is correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-14T01:14:06.207Z
Comments.Text: Since DQL/DQN is usually deep it must efficiently utilize limited experience samples possibly multiple times during any mini-batch update according to above Bellman optimality equation. Also since the NN is really minimizing a loss function so you'd have another term involving your approximated Q-function's parameter, thus you'd better need 2 different networks for these two (same) function parameters to reduce optimization complexity.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-29T09:48:48.857Z
Comments.Text: yes it is somehow correct, but it doesn't give much info about the approximation part. also, note that TD methods (as q-learning) generally do not converge to local optima in approximation. For instance, in the case of linear approximation they can converge to TD fixed point which may differ from local optima. Also, off-policy methods like Q-learning may generally diverge. So while your saying they seek to find optimal Q-value, the generally don't in the case of approximation.
Comments.UserDisplayName: Reza_va
Tags: <deep-learning><deep-rl><dqn><deep-neural-networks>
Title: What does Deep Q-Learning (DQL) do?
UserDisplayName: Ness
ThreadText: # What does Deep Q-Learning (DQL) do?
  
  (Asked by: username_0 on 2023-02-11)
  
  Hello :) I'm required to write a document where I describe what DQL does in short.
  This is what I wrote:
  
  DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a given state. Bear in mind that, we seek to find the optimal Q-value for a state-action pair at a time slot t, written as $Q^\* (s\_t,a\_t)$, which obeys the Bellman equation:
  $$Q^\*(s\_t, a\_t)=E\left [ R\_{a\_t}(s\_t,s')+\gamma max\_{a\_{t+1}}Q^\*(s\_{t+1},a\_{t+1}) \right ]$$
  
  Where $Q^\* (s\_{t+1},a\_{t+1})$ is the optimal Q-value for the next state-action pair $(s\_{t+1},a\_{t+1})$. The parameters of the DNN are updated every time step t, aiming to find the optimal policy $π^\*=argmax\_{a\_t} Q^\* (s\_t,a\_t)$ that maximizes the reward.
  
  Is what I wrote correct? Should I say `we seek to find the optimal Q-value for a state action pair at a time slot t` or should I say `we seek to train the DNN to resemble the optimal Q-value for a state action pair at a time slot t`? Which one is correct?
  
  **Tags:** deep-learning, deep-rl, dqn, deep-neural-networks
  
  > **Comments:**
  > **username_1**: Since DQL/DQN is usually deep it must efficiently utilize limited experience samples possibly multiple times during any mini-batch update according to above Bellman optimality equation. Also since the NN is really minimizing a loss function so you'd have another term involving your approximated Q-function's parameter, thus you'd better need 2 different networks for these two (same) function parameters to reduce optimization complexity.  
  > **username_2**: yes it is somehow correct, but it doesn't give much info about the approximation part. also, note that TD methods (as q-learning) generally do not converge to local optima in approximation. For instance, in the case of linear approximation they can converge to TD fixed point which may differ from local optima. Also, off-policy methods like Q-learning may generally diverge. So while your saying they seek to find optimal Q-value, the generally don't in the case of approximation.  
  
  ---
  
  ## 0 answers
  
  