Body: <p>I learned that when creating neural networks the go to was to overfit and then to regularize. However I am now in a situation where, when I make the model more complex (more layers, more filters, ...) my scores become worse.</p>
  
  <p>I am training a CNN to predict pollution 6 hours in advance. The input I give to my model is the pollution of the past 18 hours.</p>
  
  <p>Can I safely say that because there probably is a lot of noise in this data that, that is the reason when increasing my complexity, my model becomes worse?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-18T17:28:51.413Z
Comments.Text: It is mostly likely due to overfitting. What's your training loss is like? Also, do you also increase regularization effects, e.g., dropout rate, weight-decay's delta, as you make your model more complex?
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-18T18:59:43.117Z
Comments.Text: Yes it is definetly due to overfitting, but I assume that it is overfitting on noisy data, which makes it even worse. It gives quite good results on the training data, but very bad on the validation data. Even after regularizing the hell out of it, results do not become better than just a simple model with only 2 layers (in comparison to 4).
Comments.UserDisplayName: Tibo Geysen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-18T19:03:28.957Z
Comments.Text: I see. I mean if the quality of your data is low then I dont think your validation accuracy can increase no matter what you do. But then I might be wrong.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-19T04:31:44.970Z
Comments.Text: Generally for your situation I think that the noise factor in the data is keeping the validation accuracy low. Another possibility is that you do not have a sufficient data set size to fully capture the probability distribution of the data. How many training samples do you have? Also how did you select the validation data, It should be a random sample of the data set so its distribution matches that of the training set.
Comments.UserDisplayName: Gerry P
Tags: <neural-networks><convolutional-neural-networks>
Title: Why does model complexity increase my validation score by a lot?
UserDisplayName: Tibo Geysen
ThreadText: # Why does model complexity increase my validation score by a lot?
  
  (Asked by: username_0 on 2020-03-18)
  
  I learned that when creating neural networks the go to was to overfit and then to regularize. However I am now in a situation where, when I make the model more complex (more layers, more filters, ...) my scores become worse.
  
  I am training a CNN to predict pollution 6 hours in advance. The input I give to my model is the pollution of the past 18 hours.
  
  Can I safely say that because there probably is a lot of noise in this data that, that is the reason when increasing my complexity, my model becomes worse?
  
  **Tags:** neural-networks, convolutional-neural-networks
  > **Comments:**
  > **username_1**: It is mostly likely due to overfitting. What's your training loss is like? Also, do you also increase regularization effects, e.g., dropout rate, weight-decay's delta, as you make your model more complex?  
  > **username_0**: Yes it is definetly due to overfitting, but I assume that it is overfitting on noisy data, which makes it even worse. It gives quite good results on the training data, but very bad on the validation data. Even after regularizing the hell out of it, results do not become better than just a simple model with only 2 layers (in comparison to 4).  
  > **username_1**: I see. I mean if the quality of your data is low then I dont think your validation accuracy can increase no matter what you do. But then I might be wrong.  
  > **username_2**: Generally for your situation I think that the noise factor in the data is keeping the validation accuracy low. Another possibility is that you do not have a sufficient data set size to fully capture the probability distribution of the data. How many training samples do you have? Also how did you select the validation data, It should be a random sample of the data set so its distribution matches that of the training set.  
  
  ---
  
  ## 0 answers
  
  