Answers.Body: <p>There's something called <a href="https://arxiv.org/pdf/1612.00796.pdf" rel="nofollow noreferrer">Elastic Weight Consolidation</a> to prevent neural networks from forgetting previous tasks as they train on new tasks. It might be helpful for your case too.</p>
  
  <p>The main idea is to quantify the importance of parameters for task <span class="math-container">$t$</span> and penalize the model in proportion when it changes its parameters as it trains to learn task <span class="math-container">$t+1$</span>. As you can see, this incentivizes model to change parameters that are less important for task <span class="math-container">$t$</span> which prevents the model from forgetting it. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-23T23:09:45.997Z
Answers.LastActivityDate: 2020-04-23T23:09:45.997Z
Answers.UserDisplayName: SpiderRico
Body: <p>I have successfully trained a Yolo model to recognize k classes. Now I want to train by adding k+1 class to the pre-trained weights (k classes) without forgetting previous k classes. Ideally, I want to keep adding classes and train over the previous weights, i.e., train only the new classes. If I have to train all classes (k+1) every time a new class is added, it would be too time-consuming, as training k classes would take <span class="math-container">$k*20000$</span> iterations, versus the <span class="math-container">$20000$</span> iterations per new class if I can add the classes incrementally. </p>
  
  <p>The dataset is balanced (5000 images per classes for training).</p>
  
  <p>I appreciated if you can throw some methods or techniques to do this continual training for Yolo.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T17:19:03.170Z
Comments.Text: A general rule of thumb to avoid forgetting is to use a low learning rate
Comments.UserDisplayName: 0x5050
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-18T16:19:05.257Z
Comments.Text: @PradipPramanick , but lower learning rate can affect the future new class prediction accuracy right.
Comments.UserDisplayName: Troy
Tags: <convolutional-neural-networks><training><yolo><incremental-learning><catastrophic-forgetting>
Title: How can I incrementally train a Yolo model without catastrophic forgetting?
UserDisplayName: Troy
ThreadText: # How can I incrementally train a Yolo model without catastrophic forgetting?
  
  (Asked by: username_0 on 2019-07-16)
  
  I have successfully trained a Yolo model to recognize k classes. Now I want to train by adding k+1 class to the pre-trained weights (k classes) without forgetting previous k classes. Ideally, I want to keep adding classes and train over the previous weights, i.e., train only the new classes. If I have to train all classes (k+1) every time a new class is added, it would be too time-consuming, as training k classes would take $k\*20000$ iterations, versus the $20000$ iterations per new class if I can add the classes incrementally.
  
  The dataset is balanced (5000 images per classes for training).
  
  I appreciated if you can throw some methods or techniques to do this continual training for Yolo.
  
  **Tags:** convolutional-neural-networks, training, yolo, incremental-learning, catastrophic-forgetting
  
  > **Comments:**
  > **username_1**: A general rule of thumb to avoid forgetting is to use a low learning rate  
  > **username_0**: @PradipPramanick , but lower learning rate can affect the future new class prediction accuracy right.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-23. Score: 0
  
  There's something called [Elastic Weight Consolidation](https://arxiv.org/pdf/1612.00796.pdf) to prevent neural networks from forgetting previous tasks as they train on new tasks. It might be helpful for your case too.
  
  The main idea is to quantify the importance of parameters for task $t$ and penalize the model in proportion when it changes its parameters as it trains to learn task $t+1$. As you can see, this incentivizes model to change parameters that are less important for task $t$ which prevents the model from forgetting it.
  
  
  ---
  
  