Answers.Body: <p>The change in results is due to randomness. For example, your dataloader will shuffle differently the second time you run it, feeding your network slightly different batches, which results in a slightly different training run. Similarly, on network initialization, Pytorch (or any library for that matter) initializes your network with weights sampled randomly following some distribution. These weights will thus also be slightly different for different runs.</p>
  <p>As your goal is to just get the model with the lowest validation loss I recommend that, during training, you track when your model has the lowest score and just save that model. Then, when in a later epoch, your model is slightly better, you just simply save that one again. At the end, instead of using the latest model, you just use the best model from a few epochs before.</p>
  <p>If you would like to reduce the randomness of your algorithm or network, you can look into <code>seeding</code> to improve your reproducibility. Pytorch has a short <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noreferrer">page</a> on how to improve reproducibility.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-08T14:27:45.823Z
Answers.Text: Thank you. Edited in the question that the seed is also fixed, which is why it baffled me
Answers.UserDisplayName: Minh Triet
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-08T15:05:52.853Z
Answers.Text: Have you done all the things mentioned by Pytorch themselves? In addition, do you use random stuff from numpy? Do you reset seeds between runs? Do you use solely deterministic algorithms?
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-08T08:46:35.223Z
Answers.LastActivityDate: 2023-02-08T08:46:35.223Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>During a training of a neural network, the test loss was reached the minimum at the <code>x</code>-th epoch, after which I reran the training with the maximum epoch set as <code>x</code>.</p>
  <p>What is surprising to me is that the test loss in the 2nd run was very different from the 1st at epoch <code>x</code>, even though every other hyper-parameters are the same. The only change I made was the maximum epoch.</p>
  <p>What could be a possible explanation for this phenomenon?</p>
  <p>I used the standard DataLoader from Pytorch to shuffle the training data. The seed is also the same.</p>
  
Tags: <neural-networks><pytorch>
Title: Changing the number of epochs change the loss at the the `x`th epoch
UserDisplayName: Minh Triet
ThreadText: # Changing the number of epochs change the loss at the the `x`th epoch
  
  (Asked by: username_0 on 2023-02-07)
  
  During a training of a neural network, the test loss was reached the minimum at the `x`-th epoch, after which I reran the training with the maximum epoch set as `x`.
  
  What is surprising to me is that the test loss in the 2nd run was very different from the 1st at epoch `x`, even though every other hyper-parameters are the same. The only change I made was the maximum epoch.
  
  What could be a possible explanation for this phenomenon?
  
  I used the standard DataLoader from Pytorch to shuffle the training data. The seed is also the same.
  
  **Tags:** neural-networks, pytorch
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-08. Score: 0
  
  The change in results is due to randomness. For example, your dataloader will shuffle differently the second time you run it, feeding your network slightly different batches, which results in a slightly different training run. Similarly, on network initialization, Pytorch (or any library for that matter) initializes your network with weights sampled randomly following some distribution. These weights will thus also be slightly different for different runs.
  
  As your goal is to just get the model with the lowest validation loss I recommend that, during training, you track when your model has the lowest score and just save that model. Then, when in a later epoch, your model is slightly better, you just simply save that one again. At the end, instead of using the latest model, you just use the best model from a few epochs before.
  
  If you would like to reduce the randomness of your algorithm or network, you can look into `seeding` to improve your reproducibility. Pytorch has a short [page](https://pytorch.org/docs/stable/notes/randomness.html) on how to improve reproducibility.
  
  > **Comments:**
  > **username_0**: Thank you. Edited in the question that the seed is also fixed, which is why it baffled me  
  > **username_1**: Have you done all the things mentioned by Pytorch themselves? In addition, do you use random stuff from numpy? Do you reset seeds between runs? Do you use solely deterministic algorithms?  
  
  ---
  
  