Body: <p>I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture such that this is reversed, or if I definitely need to debug my dataset.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T11:06:57.467Z
Comments.Text: you mean overfitting?
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T15:11:41.580Z
Comments.Text: @Alberto Not overfitting. Validation loss never went down at all in the first place
Comments.UserDisplayName: JobHunter69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T18:23:37.397Z
Comments.Text: overfitting never says that you need the val loss to decrease initially
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T18:33:08.653Z
Comments.Text: @Alberto Then how do you know if something is overfitting vs if the validation dataset distribution is totally different than training
Comments.UserDisplayName: JobHunter69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T18:49:43.403Z
Comments.Text: you can’t, overfitting means you are not generalizing on the validation set, and by definition if the two distributions are not the same, you are not generalizing
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T20:03:51.063Z
Comments.Text: @Alberto I don't think overfitting is defined that way. I think it's defined about the model learning noise in the training dataset.
Comments.UserDisplayName: JobHunter69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T20:10:38.840Z
Comments.Text: _"the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably"_ https://en.wikipedia.org/wiki/Overfitting
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-03T21:04:28.860Z
Comments.Text: @Alberto so if i have two very different datasets, i train on dataset 1 for one epoch, the model does not fit to dataset 2. But the model does not "corresponds too closely or exactly to a particular set of data" (dataset 1) Hence it's not overfitting.
Comments.UserDisplayName: JobHunter69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-03T21:05:34.260Z
Comments.Text: @Alberto as per the definition you supplied my situation is not overfitting
Comments.UserDisplayName: JobHunter69
Tags: <neural-networks><natural-language-processing><transformer><gradient-descent>
Title: Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?
UserDisplayName: JobHunter69
ThreadText: # Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?
  
  (Asked by: username_0 on 2024-04-27)
  
  I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture such that this is reversed, or if I definitely need to debug my dataset.
  
  **Tags:** neural-networks, natural-language-processing, transformer, gradient-descent
  
  > **Comments:**
  > **username_1**: you mean overfitting?  
  > **username_0**: @username_1 Not overfitting. Validation loss never went down at all in the first place  
  > **username_1**: overfitting never says that you need the val loss to decrease initially  
  > **username_0**: @username_1 Then how do you know if something is overfitting vs if the validation dataset distribution is totally different than training  
  > **username_1**: you can’t, overfitting means you are not generalizing on the validation set, and by definition if the two distributions are not the same, you are not generalizing  
  > **username_0**: @username_1 I don't think overfitting is defined that way. I think it's defined about the model learning noise in the training dataset.  
  > **username_1**: _"the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably"_ https://en.wikipedia.org/wiki/Overfitting  
  > **username_0**: @username_1 so if i have two very different datasets, i train on dataset 1 for one epoch, the model does not fit to dataset 2. But the model does not "corresponds too closely or exactly to a particular set of data" (dataset 1) Hence it's not overfitting.  
  > **username_0**: @username_1 as per the definition you supplied my situation is not overfitting  
  
  ---
  
  ## 0 answers
  
  