Body: <p>My project include classification of images into several classes.</p>
  <p>I'm having a strange issue related to adding <a href="https://arxiv.org/abs/1710.09412" rel="nofollow noreferrer">mixup augmentation</a>. The accuracy of the training set and the validation set keep rising during all training.</p>
  <p>However, the accuracy on the test set improves significantly until epoch 80 (over baseline without mixup - grey), but when the learning rate is reduced it collapses. Note that each 40 epochs I reduce the learning rate.</p>
  <p>Is it learning rate/optimizer/scheduler problem? If so..what should I do?
  I already tried cosine schedulers and linear schedulers.</p>
  <p><a href="https://i.sstatic.net/IW9Z6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IW9Z6.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T12:09:11.620Z
Comments.Text: what dataset are you using/how large is it? Could be simply over fitting from these graphs.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T13:42:50.917Z
Comments.Text: As you can see these are validation and test graph not training graphs. Validation keep increases. Also, the grey in the test set is also increasing. Only the green (with mixup) is decreasing. The question is how to solve it
Comments.UserDisplayName: BestR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T14:14:42.460Z
Comments.Text: Your still optimizing on your validation set, so that doesn't prove it's not over fitting. And as you can see by decreasing learning rate every 40 epochs you keep fine tuning on the same batches again and again, improving slightly (cause after a while the weights basically stop changing) without any benefit on the test set, which remain slightly the same (if you smooth the scores with moving average it's most likely flat, not decreasing). Plus you have a 10 to 20% difference between the 2 sets scores, which is another strong indicator for over fitting.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T14:44:35.297Z
Comments.Text: You are right, in the grey graph the validation is basically flat because lr is low. if i dont change it varies again. But also in low lr, it improves a tiny bit. You think about early stopping? And the differences between the sets is becuase they are from different distributions.
Comments.UserDisplayName: BestR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T16:07:40.140Z
Comments.Text: well if your train, test and validation sets are different that's the very first thing to address. They are suppose to be representative of the same distribution, if not the test results are useless and not even to be interpret at all (otherwise it's like comparing apples and bananas and wondering why they don't look the same).
Comments.UserDisplayName: Edoardo Guerriero
Tags: <deep-learning><classification><training><image-processing><data-augmentation>
Title: Test accuracy go down after decreasing learning rate
UserDisplayName: BestR
ThreadText: # Test accuracy go down after decreasing learning rate
  
  (Asked by: username_0 on 2022-01-25)
  
  My project include classification of images into several classes.
  
  I'm having a strange issue related to adding [mixup augmentation](https://arxiv.org/abs/1710.09412). The accuracy of the training set and the validation set keep rising during all training.
  
  However, the accuracy on the test set improves significantly until epoch 80 (over baseline without mixup - grey), but when the learning rate is reduced it collapses. Note that each 40 epochs I reduce the learning rate.
  
  Is it learning rate/optimizer/scheduler problem? If so..what should I do?
  I already tried cosine schedulers and linear schedulers.
  
  [![enter image description here](https://i.sstatic.net/IW9Z6.png)](https://i.sstatic.net/IW9Z6.png)
  
  **Tags:** deep-learning, classification, training, image-processing, data-augmentation
  
  > **Comments:**
  > **username_1**: what dataset are you using/how large is it? Could be simply over fitting from these graphs.  
  > **username_0**: As you can see these are validation and test graph not training graphs. Validation keep increases. Also, the grey in the test set is also increasing. Only the green (with mixup) is decreasing. The question is how to solve it  
  > **username_1**: Your still optimizing on your validation set, so that doesn't prove it's not over fitting. And as you can see by decreasing learning rate every 40 epochs you keep fine tuning on the same batches again and again, improving slightly (cause after a while the weights basically stop changing) without any benefit on the test set, which remain slightly the same (if you smooth the scores with moving average it's most likely flat, not decreasing). Plus you have a 10 to 20% difference between the 2 sets scores, which is another strong indicator for over fitting.  
  > **username_0**: You are right, in the grey graph the validation is basically flat because lr is low. if i dont change it varies again. But also in low lr, it improves a tiny bit. You think about early stopping? And the differences between the sets is becuase they are from different distributions.  
  > **username_1**: well if your train, test and validation sets are different that's the very first thing to address. They are suppose to be representative of the same distribution, if not the test results are useless and not even to be interpret at all (otherwise it's like comparing apples and bananas and wondering why they don't look the same).  
  
  ---
  
  ## 0 answers
  
  