Answers.Body: <p>When doing transfer learning it makes sense to have different update policies for "inherited" parameters and the "new" parameters. "Inherited" parameters are pre-trained on dataset1 and they typically form the front end of the deep model. The "new" parameters are trained from scratch and they typically produce the desired predictions on dataset2. It would be sensible to restart the learning schedule for the "new" parameters. However, most often we would avoid doing that for "inherited" parameters in order to avoid catastrophic forgetting.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-13T10:58:12.800Z
Answers.LastActivityDate: 2019-02-13T10:58:12.800Z
Answers.UserDisplayName: ssegvic
Answers.Body: <p>Adam reduces the learning rate over time. When you change to the new training data, you want to reset the learning rate. But Adam might not be the best choice for the second round of training - it can make big changes to the inherited weights, which prevents the transfer of previous learning. It can be good to switch to simple SGD for the second round.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-15T14:31:14.633Z
Answers.LastActivityDate: 2019-03-15T14:31:14.633Z
Answers.UserDisplayName: chrishmorris
Body: <p>For example, you train on dataset 1 with an adaptive optimizer like Adam. Should you reload the learning schedule, etc., from the end of training on dataset 1 when attempting transfer to dataset 2? Why or why not?</p>
  
Tags: <neural-networks><deep-learning><transfer-learning>
Title: Should you reload the optimizer for transfer learning?
UserDisplayName: user3180
ThreadText: # Should you reload the optimizer for transfer learning?
  
  (Asked by: username_0 on 2019-02-13)
  
  For example, you train on dataset 1 with an adaptive optimizer like Adam. Should you reload the learning schedule, etc., from the end of training on dataset 1 when attempting transfer to dataset 2? Why or why not?
  
  **Tags:** neural-networks, deep-learning, transfer-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-02-13. Score: 3
  
  When doing transfer learning it makes sense to have different update policies for "inherited" parameters and the "new" parameters. "Inherited" parameters are pre-trained on dataset1 and they typically form the front end of the deep model. The "new" parameters are trained from scratch and they typically produce the desired predictions on dataset2. It would be sensible to restart the learning schedule for the "new" parameters. However, most often we would avoid doing that for "inherited" parameters in order to avoid catastrophic forgetting.
  
  
  ---
  
  ### Answer by username_2 on 2019-03-15. Score: 1
  
  Adam reduces the learning rate over time. When you change to the new training data, you want to reset the learning rate. But Adam might not be the best choice for the second round of training - it can make big changes to the inherited weights, which prevents the transfer of previous learning. It can be good to switch to simple SGD for the second round.
  
  
  ---
  
  