Answers.Body: <p>I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.</p>
  <p>First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further exploration, your dataset must contain a brunch of situations to leads your system to learn a robust policy. Imagine that you're working with robots, and training them using a fixed batch extracted using real-world interactions. If this batch was collected from a system that works pretty well, it might not contain samples about non-desirable situations. Once this system is deployed in the real world, facing a non-desirable situation, your system could not &quot;know&quot; which action should be taken to &quot;escape&quot; from these non-desirable states, once it never saw this during the learning phase. To summarizing, your dataset should be large and representative.</p>
  <p>Now we assume that your dataset is large and representative. So, another problem arises: fixed batch shift the distribution of samples, creating a kind of bias that can be extrapolated by many epochs.</p>
  <p>Finally, your dataset is composed of expert demonstrations? In other words, do you have the best actions on those samples in your dataset for each state?</p>
  <p>The good thing about a fixed batch with real-world data is that it can booster the learning process once it contains the true dynamics of the real system. But be careful regarding the distribution of samples and training hyperparameter. Resuming, there's not a recipe for all problems, it depends on your case. I'll let two references that can be useful and where this answer is based:</p>
  <p><strong>References:</strong></p>
  <p><em>Nair, A., Dalal, M., Gupta, A., &amp; Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.</em></p>
  <p><em>Fujimoto, Scott, David Meger, and Doina Precup. &quot;Off-policy deep reinforcement learning without exploration.&quot; International Conference on Machine Learning. PMLR, 2019.</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T15:33:38.827Z
Answers.LastActivityDate: 2021-02-22T15:33:38.827Z
Answers.UserDisplayName: HenDoNR
Body: <p>I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a generator to feed the data into my DQN. I've been running some automated tuning on the hyperparameters and seem to have found some good values.</p>
  <p>Now I am wondering if I should be training on the dataset more than once or whether that would cause the network to simply memorize past experiences and cause overfitting. Is there a standard practice when it comes to training on historical data in regards to the number of epochs?</p>
  <p>Edit: I'm not nessesarily looking for an answer to how many epochs I should be using, rather I'd like to know if running over the same data more than once is okay with DQNs</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-14T22:39:31.563Z
Comments.Text: The samples(experiences) store inside the experience replay those are different at each run despite of the data being same(historical data). So I don't it will matter.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-14T22:42:53.617Z
Comments.Text: @SwaksharDeb Thanks, I think I understand. Does this mean that it won't hurt the network to train more than once or that it won't benefit the network? ...or both?
Comments.UserDisplayName: Kyle Dixon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-14T22:50:03.390Z
Comments.Text: I think it will not hurt the performance and your dataset is also pretty big. If your dataset is for particular company and we are collecting experience based on past 20 years data then it is obvious thing to do.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-14T23:00:02.630Z
Comments.Text: Okay, that makes sense, I'll try it out and see how it performs, thanks. It is just for one company, IBM. The dimensions for each example are pretty large, 354 parameters, and it's got a little over 2 million examples so running through the dataset takes several hours even with a decent GPU.
Comments.UserDisplayName: Kyle Dixon
Tags: <machine-learning><reinforcement-learning><dqn><actor-critic-methods>
Title: Can I train a DQN on the same dataset for multiple epochs?
UserDisplayName: Kyle Dixon
ThreadText: # Can I train a DQN on the same dataset for multiple epochs?
  
  (Asked by: username_0 on 2021-02-14)
  
  I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a generator to feed the data into my DQN. I've been running some automated tuning on the hyperparameters and seem to have found some good values.
  
  Now I am wondering if I should be training on the dataset more than once or whether that would cause the network to simply memorize past experiences and cause overfitting. Is there a standard practice when it comes to training on historical data in regards to the number of epochs?
  
  Edit: I'm not nessesarily looking for an answer to how many epochs I should be using, rather I'd like to know if running over the same data more than once is okay with DQNs
  
  **Tags:** machine-learning, reinforcement-learning, dqn, actor-critic-methods
  
  > **Comments:**
  > **username_1**: The samples(experiences) store inside the experience replay those are different at each run despite of the data being same(historical data). So I don't it will matter.  
  > **username_0**: @username_1 Thanks, I think I understand. Does this mean that it won't hurt the network to train more than once or that it won't benefit the network? ...or both?  
  > **username_1**: I think it will not hurt the performance and your dataset is also pretty big. If your dataset is for particular company and we are collecting experience based on past 20 years data then it is obvious thing to do.  
  > **username_0**: Okay, that makes sense, I'll try it out and see how it performs, thanks. It is just for one company, IBM. The dimensions for each example are pretty large, 354 parameters, and it's got a little over 2 million examples so running through the dataset takes several hours even with a decent GPU.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-02-22. Score: 1
  
  I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.
  
  First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further exploration, your dataset must contain a brunch of situations to leads your system to learn a robust policy. Imagine that you're working with robots, and training them using a fixed batch extracted using real-world interactions. If this batch was collected from a system that works pretty well, it might not contain samples about non-desirable situations. Once this system is deployed in the real world, facing a non-desirable situation, your system could not "know" which action should be taken to "escape" from these non-desirable states, once it never saw this during the learning phase. To summarizing, your dataset should be large and representative.
  
  Now we assume that your dataset is large and representative. So, another problem arises: fixed batch shift the distribution of samples, creating a kind of bias that can be extrapolated by many epochs.
  
  Finally, your dataset is composed of expert demonstrations? In other words, do you have the best actions on those samples in your dataset for each state?
  
  The good thing about a fixed batch with real-world data is that it can booster the learning process once it contains the true dynamics of the real system. But be careful regarding the distribution of samples and training hyperparameter. Resuming, there's not a recipe for all problems, it depends on your case. I'll let two references that can be useful and where this answer is based:
  
  **References:**
  
  *Nair, A., Dalal, M., Gupta, A., & Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.*
  
  *Fujimoto, Scott, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration." International Conference on Machine Learning. PMLR, 2019.*
  
  
  ---
  
  