Answers.Body: <p>It is hard to prove because the claim is not true. The DQN algorithm is not guaranteed to converge. For a proof, and a modified algorithm C-DQN that does converge, see the <a href="https://arxiv.org/abs/2106.15419" rel="nofollow noreferrer">paper</a> by Wang and Ueda.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-15T11:41:22.117Z
Answers.LastActivityDate: 2024-12-15T11:41:22.117Z
Answers.UserDisplayName: M.C. Escher
Body: <p>Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved.</p>
  
  <p>The main difference of DQN compared to Q-Learning with linear approximator is using DNN, the experience replay memory, and the target network. Which of these components causes the issue and why?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T16:06:14.183Z
Comments.Text: See [Why doesn't Q-learning converge when using function approximation?](https://ai.stackexchange.com/q/11679/2444).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T23:25:53.340Z
Comments.Text: Thanks for the link. I carefully read the post, it does not actually answer my question.
Comments.UserDisplayName: Afshin Oroojlooy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-19T12:00:48.517Z
Comments.Text: This question is similar to: [Why doesn't Q-learning converge when using function approximation?](https://ai.stackexchange.com/questions/11679/why-doesnt-q-learning-converge-when-using-function-approximation). If you believe it’s different, please [edit] the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><dqn><convergence><function-approximation>
Title: Why is it hard to prove the convergence of the deep Q-learning algorithm?
UserDisplayName: Afshin Oroojlooy
ThreadText: # Why is it hard to prove the convergence of the deep Q-learning algorithm?
  
  (Asked by: username_0 on 2020-05-10)
  
  Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved.
  
  The main difference of DQN compared to Q-Learning with linear approximator is using DNN, the experience replay memory, and the target network. Which of these components causes the issue and why?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, convergence, function-approximation
  
  > **Comments:**
  > **username_1**: See [Why doesn't Q-learning converge when using function approximation?](https://ai.stackexchange.com/q/11679/2444).  
  > **username_0**: Thanks for the link. I carefully read the post, it does not actually answer my question.  
  > **username_1**: This question is similar to: [Why doesn't Q-learning converge when using function approximation?](https://ai.stackexchange.com/questions/11679/why-doesnt-q-learning-converge-when-using-function-approximation). If you believe it’s different, please [edit] the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-15. Score: 0
  
  It is hard to prove because the claim is not true. The DQN algorithm is not guaranteed to converge. For a proof, and a modified algorithm C-DQN that does converge, see the [paper](https://arxiv.org/abs/2106.15419) by Wang and Ueda.
  
  
  ---
  
  