<p>I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture such that this is reversed, or if I definitely need to debug my dataset.</p>

CC BY-SA 4.0
2024-04-28T11:06:57.467Z
you mean overfitting?
Alberto
CC BY-SA 4.0
2024-04-28T15:11:41.580Z
@Alberto Not overfitting. Validation loss never went down at all in the first place
JobHunter69
CC BY-SA 4.0
2024-04-28T18:23:37.397Z
overfitting never says that you need the val loss to decrease initially
Alberto
CC BY-SA 4.0
2024-04-28T18:33:08.653Z
@Alberto Then how do you know if something is overfitting vs if the validation dataset distribution is totally different than training
JobHunter69
CC BY-SA 4.0
2024-04-28T18:49:43.403Z
you can’t, overfitting means you are not generalizing on the validation set, and by definition if the two distributions are not the same, you are not generalizing
Alberto
CC BY-SA 4.0
2024-04-28T20:03:51.063Z
@Alberto I don't think overfitting is defined that way. I think it's defined about the model learning noise in the training dataset.
JobHunter69
CC BY-SA 4.0
2024-04-28T20:10:38.840Z
_"the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably"_ https://en.wikipedia.org/wiki/Overfitting
Alberto
CC BY-SA 4.0
2024-06-03T21:04:28.860Z
@Alberto so if i have two very different datasets, i train on dataset 1 for one epoch, the model does not fit to dataset 2. But the model does not "corresponds too closely or exactly to a particular set of data" (dataset 1) Hence it's not overfitting.
JobHunter69
CC BY-SA 4.0
2024-06-03T21:05:34.260Z
@Alberto as per the definition you supplied my situation is not overfitting
JobHunter69
<neural-networks><natural-language-processing><transformer><gradient-descent>
Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?
JobHunter69
# Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?

(Asked by: username_0 on 2024-04-27)

I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture such that this is reversed, or if I definitely need to debug my dataset.

**Tags:** neural-networks, natural-language-processing, transformer, gradient-descent

> **Comments:**
> **username_1**: you mean overfitting?  
> **username_0**: @username_1 Not overfitting. Validation loss never went down at all in the first place  
> **username_1**: overfitting never says that you need the val loss to decrease initially  
> **username_0**: @username_1 Then how do you know if something is overfitting vs if the validation dataset distribution is totally different than training  
> **username_1**: you can’t, overfitting means you are not generalizing on the validation set, and by definition if the two distributions are not the same, you are not generalizing  
> **username_0**: @username_1 I don't think overfitting is defined that way. I think it's defined about the model learning noise in the training dataset.  
> **username_1**: _"the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably"_ https://en.wikipedia.org/wiki/Overfitting  
> **username_0**: @username_1 so if i have two very different datasets, i train on dataset 1 for one epoch, the model does not fit to dataset 2. But the model does not "corresponds too closely or exactly to a particular set of data" (dataset 1) Hence it's not overfitting.  
> **username_0**: @username_1 as per the definition you supplied my situation is not overfitting  

---

## 0 answers

