Body: <p>I am training a ResNet model on CIFAR10 dataset. For the training subset, I selected a random 1% of the train data from the default train/test split. For the test subset I used the whole default test split (by default I mean the ones selected by the data providers). I am using cross-entropy loss, one cycle learning rate scheduler and adam optimizer. Each epoch, I am plotting accuracy of the model on both train and test subsets.
  Looking at that graph, some weird things are happening. The <strong>train accuracy achieves 100% accuracy</strong> in under 10 epochs, <strong>while the test accuracy persists to increase</strong> till the stopping epoch 60. How is the model not overfitting? One would expect that after some point during training, the model would start to overfit on train data which would result in decrease in test accuracy. In my case, it seems that the model generalizes much better if train well beyond the point where train accuracy reaches maximum.</p>
  <p>The images of train/test accuracies and losses are below:</p>
  <p><a href="https://i.sstatic.net/Bqa7j.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Bqa7j.jpg" alt="Train/test accuracies for 1% train data" /></a>
  <a href="https://i.sstatic.net/9DwiT.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9DwiT.jpg" alt="Train/test loss for 1% train data" /></a></p>
  <p>I realize that the model can still change during the training even after the accuracy on the train data reaches 100% as I am using the cross-entropy loss and with each additional epoch the predictions get more and more confident (even when all instances are already correctly classified). However, looking at the loss of the train data, it is already very close to 0 on the 20th epoch, yet still the test accuracy practically doubles after that point.</p>
  <p>What's going on, how is that possible?</p>
  <p>EDIT: Adding the results when using 100% of the train data:</p>
  <p><a href="https://i.sstatic.net/7pSuP.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7pSuP.jpg" alt="Train/test accuracy vs epoch" /></a></p>
  <p><a href="https://i.sstatic.net/rksx3.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rksx3.jpg" alt="Train/test loss vs epoch" /></a></p>
  <p>EDIT 2: Adding results after inducing 60% symmetric label noise (100% of train data used for training).</p>
  <p><a href="https://i.sstatic.net/6DHNl.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6DHNl.jpg" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/mSPXf.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mSPXf.jpg" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-19T10:05:40.423Z
Comments.Text: How many samples are in 1% of the data? What happens if you increase the training data to 50%?
Comments.UserDisplayName: BigBrownBear00
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-19T14:50:40.680Z
Comments.Text: Do you have 3 groups of data: training, dev (or validation), and test, or 2 groups of data (training and test)? You should train the model with training and dev, and then do a final assessment of model performance using the test data.
Comments.UserDisplayName: BigBrownBear00
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T09:47:16.710Z
Comments.Text: @BigBrownBear00 1% is 500 images (CIFAR10 has 50k training images). When I increase the training data to higher percentages (even when using 100%), the same thing happens, although it is not as extreme. I added two graphs for what happens when using 100% of the train data.
Comments.UserDisplayName: Liisjak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T09:48:26.543Z
Comments.Text: @BigBrownBear00 How would that change the results? My model is updating based on train data only, so having additional validation set would still result in the same train/test graphs.
Comments.UserDisplayName: Liisjak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T11:34:22.280Z
Comments.Text: I am a little confused by what you are asking. Your question states that the train data achieves 100% accuracy in under 10 epochs, but your plots don't show that. As a rule of thumb, if your training data has high accuracy but your test data does not, then by definition you are overfitting and you need to either adjust your model or add regularization. You should also try different lambda values (learning rates) to see if that helps smooth out your plots.
Comments.UserDisplayName: BigBrownBear00
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T11:39:05.300Z
Comments.Text: @BigBrownBear00 Yea, my bad. When uploading the 3rd and the 4rd image, I accidentally overwrote the 1st and the 2nd (now the original 1st and 2nd have been put them back in).
  
  Overfitting would mean that my test accuracy starts to drop after the "overfitting" point, which in my case does not happen. Futhermore, my test accuracy never stops increasing - this behaviour is the reason I posted this question.
  
  I've tried a variety of different learning rates with more or less the same results.
Comments.UserDisplayName: Liisjak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T11:52:17.230Z
Comments.Text: Any good reason why you decided to remove the paragraphs or spaces that I added? Do you think it makes your post clearer or unclearer?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T12:20:35.160Z
Comments.Text: @nbro Sorry, don't know what you're talking about.
Comments.UserDisplayName: Liisjak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T12:20:41.760Z
Comments.Text: Incorrect. High variance, or overfitting, is when you training loss is low and your test loss is high. Your plots suggest you are overfitting the data and you should look at your model or add regularization.
Comments.UserDisplayName: BigBrownBear00
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T12:33:38.967Z
Comments.Text: @BigBrownBear00 Those are typical indicators of overfitting, but do not necessarily mean that the model is overfitting. In my case, the model generalization error (error on unseen data) is decreasing constantly, meaning the model performs better and better on the unseen data, which by definition means, that the model is not yet overfitting. If the model was in fact overfitting, the generalization error would increase at some point. As an example, I added (EDIT 2) images of accuracies and losses vs epoch after inducing 60% symmetric label noise to train data. The overfitting is obvious.
Comments.UserDisplayName: Liisjak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T12:42:14.293Z
Comments.Text: I don't follow what you are you saying. Suffice to say, I don't think 500 samples is enough to train your model (your first set of plots). After you add more examples (2nd set of plots), the loss and accuracy curves follow each other more closely,  as expected. You probably need to add more information to your question: what optimizer are you using? Learning rates tried? How many layers and what does your model look like?
Comments.UserDisplayName: BigBrownBear00
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T12:51:28.950Z
Comments.Text: @BigBrownBear00 I am using Adam optimizer and one cycle learning rate scheduler with max learning rate of 0.0025 (tried 5 values between that one and 0.01 - always got very similar results). The model I'm using is ResNet9 with the same structure as described here: https://myrtle.ai/learn/how-to-train-your-resnet-4-architecture/
Comments.UserDisplayName: Liisjak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T12:53:27.387Z
Comments.Text: is your data normalized?
Comments.UserDisplayName: BigBrownBear00
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T13:00:21.167Z
Comments.Text: @BigBrownBear00 My data is normalized using the default CIFAR10 RGB normalization numbers ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)). In addition, I perform random crop and horizontal flip on the train images each epoch.
Comments.UserDisplayName: Liisjak
Tags: <machine-learning><pytorch><overfitting><cifar-10>
Title: What is wrong with my PyTorch model training on CIFAR10?
UserDisplayName: Liisjak
ThreadText: # What is wrong with my PyTorch model training on CIFAR10?
  
  (Asked by: username_0 on 2023-01-18)
  
  I am training a ResNet model on CIFAR10 dataset. For the training subset, I selected a random 1% of the train data from the default train/test split. For the test subset I used the whole default test split (by default I mean the ones selected by the data providers). I am using cross-entropy loss, one cycle learning rate scheduler and adam optimizer. Each epoch, I am plotting accuracy of the model on both train and test subsets.
  Looking at that graph, some weird things are happening. The **train accuracy achieves 100% accuracy** in under 10 epochs, **while the test accuracy persists to increase** till the stopping epoch 60. How is the model not overfitting? One would expect that after some point during training, the model would start to overfit on train data which would result in decrease in test accuracy. In my case, it seems that the model generalizes much better if train well beyond the point where train accuracy reaches maximum.
  
  The images of train/test accuracies and losses are below:
  
  [![Train/test accuracies for 1% train data](https://i.sstatic.net/Bqa7j.jpg)](https://i.sstatic.net/Bqa7j.jpg)
  [![Train/test loss for 1% train data](https://i.sstatic.net/9DwiT.jpg)](https://i.sstatic.net/9DwiT.jpg)
  
  I realize that the model can still change during the training even after the accuracy on the train data reaches 100% as I am using the cross-entropy loss and with each additional epoch the predictions get more and more confident (even when all instances are already correctly classified). However, looking at the loss of the train data, it is already very close to 0 on the 20th epoch, yet still the test accuracy practically doubles after that point.
  
  What's going on, how is that possible?
  
  EDIT: Adding the results when using 100% of the train data:
  
  [![Train/test accuracy vs epoch](https://i.sstatic.net/7pSuP.jpg)](https://i.sstatic.net/7pSuP.jpg)
  
  [![Train/test loss vs epoch](https://i.sstatic.net/rksx3.jpg)](https://i.sstatic.net/rksx3.jpg)
  
  EDIT 2: Adding results after inducing 60% symmetric label noise (100% of train data used for training).
  
  [![enter image description here](https://i.sstatic.net/6DHNl.jpg)](https://i.sstatic.net/6DHNl.jpg)
  [![enter image description here](https://i.sstatic.net/mSPXf.jpg)](https://i.sstatic.net/mSPXf.jpg)
  
  **Tags:** machine-learning, pytorch, overfitting, cifar-10
  > **Comments:**
  > **username_1**: How many samples are in 1% of the data? What happens if you increase the training data to 50%?  
  > **username_1**: Do you have 3 groups of data: training, dev (or validation), and test, or 2 groups of data (training and test)? You should train the model with training and dev, and then do a final assessment of model performance using the test data.  
  > **username_0**: @BigBrownBear00 1% is 500 images (CIFAR10 has 50k training images). When I increase the training data to higher percentages (even when using 100%), the same thing happens, although it is not as extreme. I added two graphs for what happens when using 100% of the train data.  
  > **username_0**: @BigBrownBear00 How would that change the results? My model is updating based on train data only, so having additional validation set would still result in the same train/test graphs.  
  > **username_1**: I am a little confused by what you are asking. Your question states that the train data achieves 100% accuracy in under 10 epochs, but your plots don't show that. As a rule of thumb, if your training data has high accuracy but your test data does not, then by definition you are overfitting and you need to either adjust your model or add regularization. You should also try different lambda values (learning rates) to see if that helps smooth out your plots.  
  > **username_0**: @BigBrownBear00 Yea, my bad. When uploading the 3rd and the 4rd image, I accidentally overwrote the 1st and the 2nd (now the original 1st and 2nd have been put them back in).  Overfitting would mean that my test accuracy starts to drop after the "overfitting" point, which in my case does not happen. Futhermore, my test accuracy never stops increasing - this behaviour is the reason I posted this question.  I've tried a variety of different learning rates with more or less the same results.  
  > **username_2**: Any good reason why you decided to remove the paragraphs or spaces that I added? Do you think it makes your post clearer or unclearer?  
  > **username_0**: @nbro Sorry, don't know what you're talking about.  
  > **username_1**: Incorrect. High variance, or overfitting, is when you training loss is low and your test loss is high. Your plots suggest you are overfitting the data and you should look at your model or add regularization.  
  > **username_0**: @BigBrownBear00 Those are typical indicators of overfitting, but do not necessarily mean that the model is overfitting. In my case, the model generalization error (error on unseen data) is decreasing constantly, meaning the model performs better and better on the unseen data, which by definition means, that the model is not yet overfitting. If the model was in fact overfitting, the generalization error would increase at some point. As an example, I added (EDIT 2) images of accuracies and losses vs epoch after inducing 60% symmetric label noise to train data. The overfitting is obvious.  
  > **username_1**: I don't follow what you are you saying. Suffice to say, I don't think 500 samples is enough to train your model (your first set of plots). After you add more examples (2nd set of plots), the loss and accuracy curves follow each other more closely,  as expected. You probably need to add more information to your question: what optimizer are you using? Learning rates tried? How many layers and what does your model look like?  
  > **username_0**: @BigBrownBear00 I am using Adam optimizer and one cycle learning rate scheduler with max learning rate of 0.0025 (tried 5 values between that one and 0.01 - always got very similar results). The model I'm using is ResNet9 with the same structure as described here: https://myrtle.ai/learn/how-to-train-your-resnet-4-architecture/  
  > **username_1**: is your data normalized?  
  > **username_0**: @BigBrownBear00 My data is normalized using the default CIFAR10 RGB normalization numbers ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)). In addition, I perform random crop and horizontal flip on the train images each epoch.  
  
  ---
  
  ## 0 answers
  
  