Answers.Body: <p>Transfer learning is a complete field of research, and there are multiple possibilities for what might work best in each situation.</p>
  <p>There are various ways in which you can employ a pretrained model for transfer learning. You can indeed keep the complete model intact, but it is more common (as is done in pytorch) to delete the last (several) layers. In addition, one can vary with which model weights are frozen during training. You can, for example, delete the last layer and unfreeze the second to last, meaning you train 2 layers, but one is pretrained.</p>
  <p>If you do not simply want to extend the model, it is usually advised to look into unfreezing (some of) the last pretrained layer(s), as it helps adjust the model to the new training data. Do not put the learning rate too high as it might result in the unfrozen pretrained layer forgetting what it had learned (usually termed catastrophic forgetting).</p>
  <p>I personally have had more success with deleting the last 1 or 2 layers and unfreezing another 1 or 2 layers. You could look at some academic papers to see if there are any large-scale tests for what generally works better.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-07T12:52:32.860Z
Answers.LastActivityDate: 2022-12-07T12:52:32.860Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I'm seeing conflicting info on what to do with the fully-connected output layer of a pre-trained network when it's used in transfer learning. <a href="https://datascience.stackexchange.com/questions/76370/how-many-layers-should-i-replace-in-transfer-learning-cnn">A previous answer</a> seems to imply that the network is kept intact and a new output layer is added on top of the existing output layer. This approach means that the new network will build on all of the pre-trained weights of the existing network, including the weights in the output layer. However, when I look at <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noreferrer">this tutorial</a> on PyTorch, the fully-connected output layer is replaced with a new output layer to match the number of classes in the new task. This second approach means that the pre-trained weights in the existing output layer (512*1000 parameters for ResNet18) are lost. The first approach retains everything that was learned. The second approach also looks reasonable since the weights of the final output layer are like very task-specific and so can be discarded when learning a new task. Which approach is recommended in general?</p>
  
Tags: <classification><pytorch><transfer-learning><residual-networks>
Title: Keep weights of output layer in transfer learning?
UserDisplayName: Fijoy Vadakkumpadan
ThreadText: # Keep weights of output layer in transfer learning?
  
  (Asked by: username_0 on 2022-11-26)
  
  I'm seeing conflicting info on what to do with the fully-connected output layer of a pre-trained network when it's used in transfer learning. [A previous answer](https://datascience.stackexchange.com/questions/76370/how-many-layers-should-i-replace-in-transfer-learning-cnn) seems to imply that the network is kept intact and a new output layer is added on top of the existing output layer. This approach means that the new network will build on all of the pre-trained weights of the existing network, including the weights in the output layer. However, when I look at [this tutorial](https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html) on PyTorch, the fully-connected output layer is replaced with a new output layer to match the number of classes in the new task. This second approach means that the pre-trained weights in the existing output layer (512\*1000 parameters for ResNet18) are lost. The first approach retains everything that was learned. The second approach also looks reasonable since the weights of the final output layer are like very task-specific and so can be discarded when learning a new task. Which approach is recommended in general?
  
  **Tags:** classification, pytorch, transfer-learning, residual-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-07. Score: 1
  
  Transfer learning is a complete field of research, and there are multiple possibilities for what might work best in each situation.
  
  There are various ways in which you can employ a pretrained model for transfer learning. You can indeed keep the complete model intact, but it is more common (as is done in pytorch) to delete the last (several) layers. In addition, one can vary with which model weights are frozen during training. You can, for example, delete the last layer and unfreeze the second to last, meaning you train 2 layers, but one is pretrained.
  
  If you do not simply want to extend the model, it is usually advised to look into unfreezing (some of) the last pretrained layer(s), as it helps adjust the model to the new training data. Do not put the learning rate too high as it might result in the unfrozen pretrained layer forgetting what it had learned (usually termed catastrophic forgetting).
  
  I personally have had more success with deleting the last 1 or 2 layers and unfreezing another 1 or 2 layers. You could look at some academic papers to see if there are any large-scale tests for what generally works better.
  
  
  ---
  
  