Answers.Body: <p>A learning rate of 0.1 is generally quite high for fine-tuning, especially if you're working with a pre-trained model. Fine-tuning a model usually requires a smaller learning rate (like 0.0001 to 0.001 for your case) because you don't want to disrupt the pre-trained features too quickly. You might also try using a learning rate scheduler to gradually reduce the learning rate during training along with regularization techniques.</p>
  <p>Also fine-tuning a model works well when the pre-trained model's learned features are already close to being useful for your problem. If the pre-trained model was trained on a dataset that has significantly different class distributions or domains such as including quite a few 3-class batches in your case, a fine-tuning process might not work as expected when you switch to a more balanced or different dataset. If the fine-tuning dataset is too different from the original domain-specific data, the model may &quot;forget&quot; the domain-specific features it learned in the base training phase. This is known as <a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="nofollow noreferrer">catastrophic forgetting</a>. You might need a full retrain with the class balanced training data.</p>
  <p>Finally your base model might have overfitted to the specific dataset it was trained on including the domain-specific cases. When you remove that data, the model might no longer generalize well to the new data distribution. In general the base model should be a good pretrained general-purpose foundational model, and then fine tune the model on your domain-specific dataset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-09T06:52:00.633Z
Answers.LastActivityDate: 2024-12-09T06:52:00.633Z
Answers.UserDisplayName: cinch
Body: <p>I just trained my deep learning model with a transformer for a 4-classes multi-classification problem. However, some batches only have 3 classes, making the model biased for those 3 classes and making the model not confident to infer the other class.</p>
  <p>I fine-tuned with retraining, but this time I removed that domain specific case that some batch only has 3 classes, leaving the training set and validation containing only 4 classes, and setting with higher learning rate such as <code>0.1</code> compared to the base model of <code>0.001</code>.</p>
  <p>But why is my fine-tuned model not good? Its accuracy is about 60%-80% compared to base-model that has 90% accuracy.</p>
  
Tags: <deep-learning><fine-tuning><accuracy><testing><learning-rate>
Title: In some case, is training from random weight initialization better than fine-tuning?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # In some case, is training from random weight initialization better than fine-tuning?
  
  (Asked by: username_0 on 2024-12-08)
  
  I just trained my deep learning model with a transformer for a 4-classes multi-classification problem. However, some batches only have 3 classes, making the model biased for those 3 classes and making the model not confident to infer the other class.
  
  I fine-tuned with retraining, but this time I removed that domain specific case that some batch only has 3 classes, leaving the training set and validation containing only 4 classes, and setting with higher learning rate such as `0.1` compared to the base model of `0.001`.
  
  But why is my fine-tuned model not good? Its accuracy is about 60%-80% compared to base-model that has 90% accuracy.
  
  **Tags:** deep-learning, fine-tuning, accuracy, testing, learning-rate
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-09. Score: 1 | Accepted answer
  
  A learning rate of 0.1 is generally quite high for fine-tuning, especially if you're working with a pre-trained model. Fine-tuning a model usually requires a smaller learning rate (like 0.0001 to 0.001 for your case) because you don't want to disrupt the pre-trained features too quickly. You might also try using a learning rate scheduler to gradually reduce the learning rate during training along with regularization techniques.
  
  Also fine-tuning a model works well when the pre-trained model's learned features are already close to being useful for your problem. If the pre-trained model was trained on a dataset that has significantly different class distributions or domains such as including quite a few 3-class batches in your case, a fine-tuning process might not work as expected when you switch to a more balanced or different dataset. If the fine-tuning dataset is too different from the original domain-specific data, the model may "forget" the domain-specific features it learned in the base training phase. This is known as [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference). You might need a full retrain with the class balanced training data.
  
  Finally your base model might have overfitted to the specific dataset it was trained on including the domain-specific cases. When you remove that data, the model might no longer generalize well to the new data distribution. In general the base model should be a good pretrained general-purpose foundational model, and then fine tune the model on your domain-specific dataset.
  
  
  ---
  
  