Body: <p>I want to train my vision transformer model on a benchmark for an image segmentation task: (<a href="https://arxiv.org/abs/2110.08733" rel="nofollow noreferrer">LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation</a>) (<a href="https://github.com/Junjue-Wang/LoveDA" rel="nofollow noreferrer">GitHub</a>), but I don't get any acceptable result from my model.</p>
  <p>I don't have a good result either in evaluation nor (obviously) in test.</p>
  <p>I'm frustrated since I have trained my model with different parameters: learning rate : [0.001,0001, 6e-e].</p>
  <p>The training datasets include about 3000 images, the number of epochs is 1500.</p>
  <p>I have used different loss functions: Cross Entropy, Dice loss, Focal loss, and a combination of those losses.</p>
  <p>So, I'm wondering if anyone has any experience that could help me find a solution and improve my model's performance?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-08T14:48:08.070Z
Comments.Text: How does the loss function behave during training?
Comments.UserDisplayName: Lee Reeves
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-08T16:35:59.890Z
Comments.Text: thanks Lee,
  well cross entropy works better than any of them it decreases gradually, but after about 1200 epochs it will stop decreasing
  Also I have checked the model with more general dataset, it was working fine 
  but still have difficulty in compareto other deep learning like deeplab
Comments.UserDisplayName: sara yaghoobi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-08T16:55:46.990Z
Comments.Text: @sarayaghoobi Edit your post to include the plots of the loss and other metrics, if possible. Said that, a training dataset of 3000 images seems to be very small. How many training parameters does your model have?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-08T17:06:12.080Z
Comments.Text: Have you tried a learning rate schedule? A lower learning rate might help the model fine tune after converging at a poor solution.
Comments.UserDisplayName: Lee Reeves
Tags: <training><transformer><image-segmentation><testing>
Title: What considerations should I take to train my transformer model?
UserDisplayName: sara yaghoobi
ThreadText: # What considerations should I take to train my transformer model?
  
  (Asked by: username_0 on 2022-06-08)
  
  I want to train my vision transformer model on a benchmark for an image segmentation task: ([LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation](https://arxiv.org/abs/2110.08733)) ([GitHub](https://github.com/Junjue-Wang/LoveDA)), but I don't get any acceptable result from my model.
  
  I don't have a good result either in evaluation nor (obviously) in test.
  
  I'm frustrated since I have trained my model with different parameters: learning rate : [0.001,0001, 6e-e].
  
  The training datasets include about 3000 images, the number of epochs is 1500.
  
  I have used different loss functions: Cross Entropy, Dice loss, Focal loss, and a combination of those losses.
  
  So, I'm wondering if anyone has any experience that could help me find a solution and improve my model's performance?
  
  **Tags:** training, transformer, image-segmentation, testing
  
  > **Comments:**
  > **username_1**: How does the loss function behave during training?  
  > **username_0**: thanks Lee, well cross entropy works better than any of them it decreases gradually, but after about 1200 epochs it will stop decreasing Also I have checked the model with more general dataset, it was working fine  but still have difficulty in compareto other deep learning like deeplab  
  > **username_2**: @username_0 Edit your post to include the plots of the loss and other metrics, if possible. Said that, a training dataset of 3000 images seems to be very small. How many training parameters does your model have?  
  > **username_1**: Have you tried a learning rate schedule? A lower learning rate might help the model fine tune after converging at a poor solution.  
  
  ---
  
  ## 0 answers
  
  