Answers.Body: <p>Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same reason the validation accuracy increases: learning less noise means better generalization on unseen data. These two aspects are sides of the same coin.</p>
  <p>There would be no point in developing a technique that only reduce training accuracy to match for aesthetic reasons the validation one.
  So yes, you can keep investigating regularization techniques and you should always use them. But be aware that since you have a strict goal you want to achieve you might also want to try other architectures or expand your dataset. Regularization will mostly help you to get a better grasp on the best scores you can achieve (if you get 84% accuracy on training, I argue you'll probably never get to 90% validation, unless the model was far from converging) and they boost a bit good results, but they will not help you making a huge jump up.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-29T07:54:00.150Z
Answers.LastActivityDate: 2022-07-29T07:54:00.150Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset.</p>
  <p>Using some classical regularization techniques (dropout=0.5) and weight_decay=0.7, I have an accuracy of 84% on the training dataset and 70% on the validation dataset.</p>
  <p>My question is: Do regularization techniques usually improve the validation accuracy? Or they will only reduce the training accuracy to a closer level to the validation accuracy?</p>
  <p>As my objective is improve the validation accuracy to 90%, and I am wondering if there is any hope that solving the overfitting problem would increase validation accuracy, so should I continue investigating on the regularization techniques or thinking about changing the whole model.</p>
  
Tags: <neural-networks><overfitting><bert>
Title: fondamental question about regularization techniques to solve overfitting problem in neural networks
UserDisplayName: tammuz
ThreadText: # fondamental question about regularization techniques to solve overfitting problem in neural networks
  
  (Asked by: username_0 on 2022-07-28)
  
  I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset.
  
  Using some classical regularization techniques (dropout=0.5) and weight\_decay=0.7, I have an accuracy of 84% on the training dataset and 70% on the validation dataset.
  
  My question is: Do regularization techniques usually improve the validation accuracy? Or they will only reduce the training accuracy to a closer level to the validation accuracy?
  
  As my objective is improve the validation accuracy to 90%, and I am wondering if there is any hope that solving the overfitting problem would increase validation accuracy, so should I continue investigating on the regularization techniques or thinking about changing the whole model.
  
  **Tags:** neural-networks, overfitting, bert
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-29. Score: 2 | Accepted answer
  
  Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same reason the validation accuracy increases: learning less noise means better generalization on unseen data. These two aspects are sides of the same coin.
  
  There would be no point in developing a technique that only reduce training accuracy to match for aesthetic reasons the validation one.
  So yes, you can keep investigating regularization techniques and you should always use them. But be aware that since you have a strict goal you want to achieve you might also want to try other architectures or expand your dataset. Regularization will mostly help you to get a better grasp on the best scores you can achieve (if you get 84% accuracy on training, I argue you'll probably never get to 90% validation, unless the model was far from converging) and they boost a bit good results, but they will not help you making a huge jump up.
  
  
  ---
  
  