Answers.Body: <p>from your post I assume you are having three sub-questions and I will answer them one by one.</p>
  <ul>
  <li>For the 1st question, yes, layer freezing reduces the computational cost a lot and also helps the model to keep the many patterns it has learned to recognize things better, c negative to positive samples easier and avoid overfitting if it was trained on a bigger dataset. Said like having a model trained on COCO but now we only want a person detection model, without freezing weights, our model may not generalize as well as it was before due to backprops now just focusing on one class.</li>
  <li>The second one, yes! Without freezing but still using pretrained model as initial weights is also counted as transfer learning</li>
  <li>Finally, in summary, layer freezing is a very essential trick to have a good model, especially if you want it to work well in the open-world dataset and generalize for as many domains as possible, time-saving and easier for fine-tuning as we now just have to train the final downstream task layers.
  Hope this helps :D</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T18:37:13.743Z
Answers.LastActivityDate: 2023-02-21T18:37:13.743Z
Answers.UserDisplayName: GunFire
Answers.Body: <p>Both are transfer learning approaches, which this Pytorch tutorial explains very well:</p>
  <p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noreferrer">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
  <blockquote>
  <p>In practice, very few people train an entire Convolutional Network
  from scratch (with random initialization), because it is relatively
  rare to have a dataset of sufficient size. Instead, it is common to
  pretrain a ConvNet on a very large dataset (e.g. ImageNet, which
  contains 1.2 million images with 1000 categories), and then use the
  ConvNet either as an initialization or a fixed feature extractor for
  the task of interest.</p>
  <p>These two major transfer learning scenarios look as follows:</p>
  <p><strong>Finetuning the convnet</strong>: Instead of random initialization, we
  initialize the network with a pretrained network, like the one that is
  trained on imagenet 1000 dataset. Rest of the training looks as usual.</p>
  <p><strong>ConvNet as fixed feature extractor</strong>: Here, we will freeze the weights
  for all of the network except that of the final fully connected layer.
  This last fully connected layer is replaced with a new one with random
  weights and only this layer is trained.</p>
  </blockquote>
  <p>In summary layer freezing is faster, but it is less accurate (after enough training) than finetuning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T21:08:30.240Z
Answers.LastActivityDate: 2023-02-21T21:08:30.240Z
Answers.UserDisplayName: Rexcirus
Body: <p>In Deep Learning and Transfer Learning, does layer freezing offer other benefits other than to reduce computational time in gradient descent?</p>
  <p>Assuming I train a neural network on task A to derive weights <span class="math-container">$W_{A}$</span>, set these as initial weights and train on another task B (without layer freezing), does it still count as transfer learning?</p>
  <p>In summary, how essential is layer freezing in transfer learning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-23T09:49:49.050Z
Comments.Text: Welcome to this SE :) Be sure to look around on the SE for all the questions that others have posted. The following describes some of the things you are asking for: https://ai.stackexchange.com/questions/22963/what-is-layer-freezing-in-transfer-learning
Comments.UserDisplayName: Robin van Hoorn
Tags: <deep-learning><training><transfer-learning><fine-tuning>
Title: Does layer freezing offer other benefits other than to reduce computational time in gradient descent?
UserDisplayName: Enk9456
ThreadText: # Does layer freezing offer other benefits other than to reduce computational time in gradient descent?
  
  (Asked by: username_0 on 2023-02-21)
  
  In Deep Learning and Transfer Learning, does layer freezing offer other benefits other than to reduce computational time in gradient descent?
  
  Assuming I train a neural network on task A to derive weights $W\_{A}$, set these as initial weights and train on another task B (without layer freezing), does it still count as transfer learning?
  
  In summary, how essential is layer freezing in transfer learning?
  
  **Tags:** deep-learning, training, transfer-learning, fine-tuning
  > **Comments:**
  > **username_1**: Welcome to this SE :) Be sure to look around on the SE for all the questions that others have posted. The following describes some of the things you are asking for: https://ai.stackexchange.com/questions/22963/what-is-layer-freezing-in-transfer-learning  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-02-21. Score: 1
  
  from your post I assume you are having three sub-questions and I will answer them one by one.
  
  * For the 1st question, yes, layer freezing reduces the computational cost a lot and also helps the model to keep the many patterns it has learned to recognize things better, c negative to positive samples easier and avoid overfitting if it was trained on a bigger dataset. Said like having a model trained on COCO but now we only want a person detection model, without freezing weights, our model may not generalize as well as it was before due to backprops now just focusing on one class.
  * The second one, yes! Without freezing but still using pretrained model as initial weights is also counted as transfer learning
  * Finally, in summary, layer freezing is a very essential trick to have a good model, especially if you want it to work well in the open-world dataset and generalize for as many domains as possible, time-saving and easier for fine-tuning as we now just have to train the final downstream task layers.
    Hope this helps :D
  
  
  ---
  
  ### Answer by username_3 on 2023-02-21. Score: 1
  
  Both are transfer learning approaches, which this Pytorch tutorial explains very well:
  
  <https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html>
  
  > In practice, very few people train an entire Convolutional Network
  > from scratch (with random initialization), because it is relatively
  > rare to have a dataset of sufficient size. Instead, it is common to
  > pretrain a ConvNet on a very large dataset (e.g. ImageNet, which
  > contains 1.2 million images with 1000 categories), and then use the
  > ConvNet either as an initialization or a fixed feature extractor for
  > the task of interest.
  >
  > These two major transfer learning scenarios look as follows:
  >
  > **Finetuning the convnet**: Instead of random initialization, we
  > initialize the network with a pretrained network, like the one that is
  > trained on imagenet 1000 dataset. Rest of the training looks as usual.
  >
  > **ConvNet as fixed feature extractor**: Here, we will freeze the weights
  > for all of the network except that of the final fully connected layer.
  > This last fully connected layer is replaced with a new one with random
  > weights and only this layer is trained.
  
  In summary layer freezing is faster, but it is less accurate (after enough training) than finetuning.
  
  
  ---
  
  