Body: <p>I am training a segmentation model on 3D data, after around 170 epochs which took around 4 days, I notice the model is no more learning and the dice score is at 0.51. What is the best approach at this point to keep model learning?<br />
  Learning rate: 1e-4<br />
  Batch size: 6<br />
  optimizer: adamw<br />
  loss: generalized dice loss<br />
  PS: there are more augmentation in training data than validation data, this might be the reason, the validation loss curve is below training loss curve</p>
  <p><a href="https://i.sstatic.net/DqP9n.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DqP9n.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/BxHVa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BxHVa.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-27T06:57:01.090Z
Comments.Text: There are hundreds of things you could try: Muck around with hyper-parameters, use a learning rate scheduler, change the model structure, more augmentation, less augmentation. It's just a matter of experimentation. I'd suggest using a grid search on a smaller sample that doesn't take 4 days to train, then use those values on the real training set.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-27T11:28:10.917Z
Comments.Text: Additional to @Recessive's response, very specifically, go for smaller learning rate. As he suggested, you can use learning rate scheduler for that, which will reduce learning rate as epochs pass.
Comments.UserDisplayName: meliksahturker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-27T14:37:33.130Z
Comments.Text: i already tried reduce on plateau but tht did not helped much
Comments.UserDisplayName: Talha Anwar
Tags: <machine-learning><deep-learning><semantic-segmentation>
Title: What to do when model stops learning after some epochs
UserDisplayName: Talha Anwar
ThreadText: # What to do when model stops learning after some epochs
  
  (Asked by: username_0 on 2021-07-26)
  
  I am training a segmentation model on 3D data, after around 170 epochs which took around 4 days, I notice the model is no more learning and the dice score is at 0.51. What is the best approach at this point to keep model learning?  
  Learning rate: 1e-4  
  Batch size: 6  
  optimizer: adamw  
  loss: generalized dice loss  
  PS: there are more augmentation in training data than validation data, this might be the reason, the validation loss curve is below training loss curve
  
  [![enter image description here](https://i.sstatic.net/DqP9n.png)](https://i.sstatic.net/DqP9n.png)
  [![enter image description here](https://i.sstatic.net/BxHVa.png)](https://i.sstatic.net/BxHVa.png)
  
  **Tags:** machine-learning, deep-learning, semantic-segmentation
  > **Comments:**
  > **username_1**: There are hundreds of things you could try: Muck around with hyper-parameters, use a learning rate scheduler, change the model structure, more augmentation, less augmentation. It's just a matter of experimentation. I'd suggest using a grid search on a smaller sample that doesn't take 4 days to train, then use those values on the real training set.  
  > **username_2**: Additional to @Recessive's response, very specifically, go for smaller learning rate. As he suggested, you can use learning rate scheduler for that, which will reduce learning rate as epochs pass.  
  > **username_0**: i already tried reduce on plateau but tht did not helped much  
  
  ---
  
  ## 0 answers
  
  