Body: <p>I have a large data set that doesn't fit in memory and would have to use something like Keras's <code>model.fit_generator</code> if I would like to train the model on all of the available data. The problem is that my data load time is larger than a single epoch and I would hate to incur that data load cost for each epoch.</p>
  
  <p>The alternative approach that yields some value is to load as much data as possible, train the model for a few hundred epochs, then load the next portion of the data and reiterate for the same amount of epochs. And repeat this until all my data is "seen" by the model. </p>
  
  <p>Intuitively I understand that this is sub-optimal as the model will tend to optimize for the latest portion of the data and "forget" the previous data but I would like a more in-depth explanation of the downsides of that method and if there are any ways to overcome them.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-11T16:51:22.813Z
Comments.Text: What do you mean by "data load time"? Do you mean the time required to load a batch of data or to load the whole dataset?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-12T07:17:09.373Z
Comments.Text: @nbro an epoch on 20K samples, with 32 mini batch size, takes about 60 seconds on a 1080ti with my simple model. loading those 20K samples takes about 120 seconds. I have 200K samples. the problem is a spatial perception form images (if that makes any difference)
Comments.UserDisplayName: Георги Кременлиев
Tags: <training><datasets><keras>
Title: Difference between retraining on different portions of data and training initially on larger data set
UserDisplayName: Георги Кременлиев
ThreadText: # Difference between retraining on different portions of data and training initially on larger data set
  
  (Asked by: username_0 on 2019-05-10)
  
  I have a large data set that doesn't fit in memory and would have to use something like Keras's `model.fit_generator` if I would like to train the model on all of the available data. The problem is that my data load time is larger than a single epoch and I would hate to incur that data load cost for each epoch.
  
  The alternative approach that yields some value is to load as much data as possible, train the model for a few hundred epochs, then load the next portion of the data and reiterate for the same amount of epochs. And repeat this until all my data is "seen" by the model.
  
  Intuitively I understand that this is sub-optimal as the model will tend to optimize for the latest portion of the data and "forget" the previous data but I would like a more in-depth explanation of the downsides of that method and if there are any ways to overcome them.
  
  **Tags:** training, datasets, keras
  > **Comments:**
  > **username_1**: What do you mean by "data load time"? Do you mean the time required to load a batch of data or to load the whole dataset?  
  > **username_0**: @nbro an epoch on 20K samples, with 32 mini batch size, takes about 60 seconds on a 1080ti with my simple model. loading those 20K samples takes about 120 seconds. I have 200K samples. the problem is a spatial perception form images (if that makes any difference)  
  
  ---
  
  ## 0 answers
  
  