Answers.Body: <p>In <strong>rehearsal</strong>, you do <em>not</em> necessarily train with <em>all</em> old training data, but you can just use <em>some</em> of it [<a href="https://www.tandfonline.com/doi/pdf/10.1080/09540099550039318?casa_token=HhD-UUb3VhkAAAAA:rveRjk5kTKF1dXwuOBsZoAkEfwD6NCQ8axBAtvQB0KONwXdaYbyH5tsfn-KkmbwIk4R2pECS0JM5" rel="nofollow noreferrer">1</a>], which you add to your current (or new) training data.</p>
  <p>In <strong>batch learning</strong>, at every epoch, you typically train with all training data, every step with a different batch (or subset) of the training data; so, if you have <span class="math-container">$N$</span> training examples and your batch size is <span class="math-container">$M$</span>, then you will have <span class="math-container">$\lfloor N/M \rfloor $</span> gradient descent steps.</p>
  <p>There is also <strong>pseudo-rehearsal</strong>, which is an alternative to rehearsal that is useful when you may not have access to the previously used training data [<a href="https://www.tandfonline.com/doi/pdf/10.1080/09540099550039318?casa_token=HhD-UUb3VhkAAAAA:rveRjk5kTKF1dXwuOBsZoAkEfwD6NCQ8axBAtvQB0KONwXdaYbyH5tsfn-KkmbwIk4R2pECS0JM5" rel="nofollow noreferrer">1</a>] (for example, because it is expensive to store).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-11T12:29:10.903Z
Answers.Text: TODO: I really need to explain what "pseudo-rehearsal" is, which I don't do it now.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-29T19:43:40.543Z
Answers.LastActivityDate: 2020-09-29T19:43:40.543Z
Answers.UserDisplayName: nbro
Body: <p>I am learning about <em>incremental learning</em> and read that <em>rehearsal</em> learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? You train a model by passing in batches of data and redo this with a set number of epochs.</p>
  <p>If I'm understanding rehearsal learning correctly, you do the exact same thing but with &quot;new&quot; data. Thus, the only difference is inconsistencies in the epoch number across data batches.</p>
  
Tags: <neural-networks><deep-learning><gradient-descent><incremental-learning><batch-learning>
Title: Is batch learning with gradient descent equivalent to "rehearsal" in incremental learning?
UserDisplayName: JobHunter69
ThreadText: # Is batch learning with gradient descent equivalent to "rehearsal" in incremental learning?
  
  (Asked by: username_0 on 2020-09-07)
  
  I am learning about *incremental learning* and read that *rehearsal* learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? You train a model by passing in batches of data and redo this with a set number of epochs.
  
  If I'm understanding rehearsal learning correctly, you do the exact same thing but with "new" data. Thus, the only difference is inconsistencies in the epoch number across data batches.
  
  **Tags:** neural-networks, deep-learning, gradient-descent, incremental-learning, batch-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-29. Score: 0
  
  In **rehearsal**, you do *not* necessarily train with *all* old training data, but you can just use *some* of it [[1](https://www.tandfonline.com/doi/pdf/10.1080/09540099550039318?casa_token=HhD-UUb3VhkAAAAA:rveRjk5kTKF1dXwuOBsZoAkEfwD6NCQ8axBAtvQB0KONwXdaYbyH5tsfn-KkmbwIk4R2pECS0JM5)], which you add to your current (or new) training data.
  
  In **batch learning**, at every epoch, you typically train with all training data, every step with a different batch (or subset) of the training data; so, if you have $N$ training examples and your batch size is $M$, then you will have $\lfloor N/M \rfloor $ gradient descent steps.
  
  There is also **pseudo-rehearsal**, which is an alternative to rehearsal that is useful when you may not have access to the previously used training data [[1](https://www.tandfonline.com/doi/pdf/10.1080/09540099550039318?casa_token=HhD-UUb3VhkAAAAA:rveRjk5kTKF1dXwuOBsZoAkEfwD6NCQ8axBAtvQB0KONwXdaYbyH5tsfn-KkmbwIk4R2pECS0JM5)] (for example, because it is expensive to store).
  
  > **Comments:**
  > **username_1**: TODO: I really need to explain what "pseudo-rehearsal" is, which I don't do it now.  
  
  ---
  
  