Answers.Body: <p>It is not so much the problem of using Reinforcement Learning to train the neural networks, it is the assumptions made about the data given to standard Neural Networks. They are not capable of handling strongly correlated data which is one of the motivations for introducing Recurrent Neural Networks, as they can handle this correlated data well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-23T14:01:53.993Z
Answers.LastActivityDate: 2020-07-23T14:01:53.993Z
Answers.UserDisplayName: David
Body: <p>While reading <a href="https://arxiv.org/pdf/1312.5602.pdf" rel="nofollow noreferrer">the DQN paper</a>, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network).</p>
  <p>So, why does Reinforcement Learning using a non-linear function approximator diverge when using strongly correlated data as input?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-11T08:19:01.937Z
Comments.Text: Read chapter 11 of [this](http://incompleteideas.net/book/bookdraft2017nov5.pdf) book. This is only a draft, if you can find the full book even better. Also, I think similar questions were answered already so try searching a bit through the website.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-11T16:05:18.390Z
Comments.Text: Maybe this is a duplicate of [Why doesn't Q-learning converge when using function approximation?](https://ai.stackexchange.com/q/11679/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><dqn><deep-rl><convergence><function-approximation>
Title: Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?
UserDisplayName: 강문주
ThreadText: # Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?
  
  (Asked by: username_0 on 2020-01-29)
  
  While reading [the DQN paper](https://arxiv.org/pdf/1312.5602.pdf), I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network).
  
  So, why does Reinforcement Learning using a non-linear function approximator diverge when using strongly correlated data as input?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, convergence, function-approximation
  
  > **Comments:**
  > **username_1**: Read chapter 11 of [this](http://incompleteideas.net/book/bookdraft2017nov5.pdf) book. This is only a draft, if you can find the full book even better. Also, I think similar questions were answered already so try searching a bit through the website.  
  > **username_2**: Maybe this is a duplicate of [Why doesn't Q-learning converge when using function approximation?](https://ai.stackexchange.com/q/11679/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-07-23. Score: 2
  
  It is not so much the problem of using Reinforcement Learning to train the neural networks, it is the assumptions made about the data given to standard Neural Networks. They are not capable of handling strongly correlated data which is one of the motivations for introducing Recurrent Neural Networks, as they can handle this correlated data well.
  
  
  ---
  
  