Answers.Body: <p>This you will usually face a lot and according to my experience, it is not really abnormal.</p>
  <p>One of the major reasons for this is due to the training loss being calculated during training for each epoch, while validation loss is calculated after the training is done, so the model already has previous knowledge about what kind of data you are using and essentially its distribution.</p>
  <p>The second reason could be if your dense layers are regularized (as I see you are using L1 dropout), this regularization happens only during training. So the training loss kinda compensates for a better validation loss (this is the job of regularization).</p>
  <p>Also I see that your validation set is only 15% of the total data, you can increase this percentage to about 20%.</p>
  <p>One of my favorite ML <a href="https://rads.stackoverflow.com/amzn/click/com/1098125975" rel="nofollow noreferrer" rel="nofollow noreferrer">books</a> author Aurelien Geron explained all these in a <a href="https://twitter.com/aureliengeron/status/1110839345609465856" rel="nofollow noreferrer">thread</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T16:42:58.210Z
Answers.Text: thanks for the help! much appreciated, i will try to increase my validation set and also check the val_loss at the start of the training
Answers.UserDisplayName: Roger Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T13:31:49.853Z
Answers.LastActivityDate: 2024-01-10T13:31:49.853Z
Answers.UserDisplayName: Suvo
Body: <p>I've been training several types of MLPs with different optimisers and tuned them with keras's hyperband tuner. All of them follow this cone architecture:
  <a href="https://i.sstatic.net/slf6R.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/slf6R.jpg" alt="general representation of the model" /></a></p>
  <p>All the networks were trained on the same dataset composed of 342k input examples and another different 62k examples for the validation (aprox 20% ratio). No example is shared between them. The data come from the same distribution :</p>
  <p>KstestResult(statistic=0.06393057115770212, pvalue=1.8775454622663613e-165, statistic_location=-0.045045708259117134, statistic_sign=1)</p>
  <p>Whatever i try, the validation loss starts below the training one as show below in the graphs (the val_loss in the graphs is shifted by 0.5epch to account for the calculation of train_loss):</p>
  <p>1st: Adadelta and mae
  <a href="https://i.sstatic.net/cZMk1.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cZMk1.jpg" alt="gph1" /></a>
  2nd: Adam and mse
  <a href="https://i.sstatic.net/8RId1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8RId1.png" alt="gph4" /></a>
  3rd: Adadelta and mse
  <a href="https://i.sstatic.net/xurmv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xurmv.png" alt="gph3" /></a>
  4th: Adam and mse + L1 Dropout
  <a href="https://i.sstatic.net/RWMBD.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RWMBD.jpg" alt="gph2" /></a></p>
  <p>All the models have very similar rmse and accuracy and are almost usable, the 3rd one also performing the best out of the bunch. I'm curious why does the val_loss always start below the trainig one, and if this is a problem since it would technically mean that i'm overfitting the model from the start :-? Also what could i do to further investigate the issue and general advice. Thanks in advance for the replies!</p>
  
Tags: <training><multilayer-perceptrons><validation-loss>
Title: Validation loss is always lower than training loss whatever i try
UserDisplayName: Roger Smith
ThreadText: # Validation loss is always lower than training loss whatever i try
  
  (Asked by: username_0 on 2024-01-10)
  
  I've been training several types of MLPs with different optimisers and tuned them with keras's hyperband tuner. All of them follow this cone architecture:
  [![general representation of the model](https://i.sstatic.net/slf6R.jpg)](https://i.sstatic.net/slf6R.jpg)
  
  All the networks were trained on the same dataset composed of 342k input examples and another different 62k examples for the validation (aprox 20% ratio). No example is shared between them. The data come from the same distribution :
  
  KstestResult(statistic=0.06393057115770212, pvalue=1.8775454622663613e-165, statistic\_location=-0.045045708259117134, statistic\_sign=1)
  
  Whatever i try, the validation loss starts below the training one as show below in the graphs (the val\_loss in the graphs is shifted by 0.5epch to account for the calculation of train\_loss):
  
  1st: Adadelta and mae
  [![gph1](https://i.sstatic.net/cZMk1.jpg)](https://i.sstatic.net/cZMk1.jpg)
  2nd: Adam and mse
  [![gph4](https://i.sstatic.net/8RId1.png)](https://i.sstatic.net/8RId1.png)
  3rd: Adadelta and mse
  [![gph3](https://i.sstatic.net/xurmv.png)](https://i.sstatic.net/xurmv.png)
  4th: Adam and mse + L1 Dropout
  [![gph2](https://i.sstatic.net/RWMBD.jpg)](https://i.sstatic.net/RWMBD.jpg)
  
  All the models have very similar rmse and accuracy and are almost usable, the 3rd one also performing the best out of the bunch. I'm curious why does the val\_loss always start below the trainig one, and if this is a problem since it would technically mean that i'm overfitting the model from the start :-? Also what could i do to further investigate the issue and general advice. Thanks in advance for the replies!
  
  **Tags:** training, multilayer-perceptrons, validation-loss
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-10. Score: 1
  
  This you will usually face a lot and according to my experience, it is not really abnormal.
  
  One of the major reasons for this is due to the training loss being calculated during training for each epoch, while validation loss is calculated after the training is done, so the model already has previous knowledge about what kind of data you are using and essentially its distribution.
  
  The second reason could be if your dense layers are regularized (as I see you are using L1 dropout), this regularization happens only during training. So the training loss kinda compensates for a better validation loss (this is the job of regularization).
  
  Also I see that your validation set is only 15% of the total data, you can increase this percentage to about 20%.
  
  One of my favorite ML [books](https://rads.stackoverflow.com/amzn/click/com/1098125975) author Aurelien Geron explained all these in a [thread](https://twitter.com/aureliengeron/status/1110839345609465856).
  
  > **Comments:**
  > **username_0**: thanks for the help! much appreciated, i will try to increase my validation set and also check the val_loss at the start of the training  
  
  ---
  
  