Answers.Body: <p>You have a small dataset.  Should you even be using neural nets?  Have you done any diagnostics to see if you even have enough data?  Are you using the right metric?  Accuracy is not always the correct metric.  Which weights are you retaining?  You will overfit if you save the weights that produce the lowest training error.  Save the weights that produced the lowest validation error.  L1, L2, and dropout are all great.  So many things not described in the problem...</p>
  
  <p><a href="http://www.ultravioletanalytics.com/blog/kaggle-titanic-competition-part-ix-bias-variance-and-learning-curves" rel="nofollow noreferrer">http://www.ultravioletanalytics.com/blog/kaggle-titanic-competition-part-ix-bias-variance-and-learning-curves</a></p>
  
  <p>I'm wondering why you're not trying interpretable models to see if the resulting weights for the features make sense.  Also if your comparing all those models and parameters, set your random initial starting point to be the same by setting the seed.  I also hope you are using the same training set for each model.</p>
  
  <p>You probably need more data...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-20T21:44:31.957Z
Answers.LastActivityDate: 2019-07-20T21:44:31.957Z
Answers.UserDisplayName: MrStutterZ
Answers.Body: <p>Your network without regularization does not appear to be over fitting but rather it appears to be converging to a minima. I am actually a bit surprised it is doing as well as it is given that your data set is small. So You don't need regularization. If you want to improve the accuracy you might try using an adjustable learning rate. The Keras call back ReduceLROnPlateau can be used for this. Documentation is <a href="https://keras.io/api/callbacks/reduce_lr_on_plateau/" rel="nofollow noreferrer">here</a>. Also use the callback ModelCheckpoint to save the model with the lowest validation loss. Documentation is <a href="https://keras.io/api/callbacks/model_checkpoint/" rel="nofollow noreferrer">here</a>. It would help a lot if you posted your model code. I have found if you do encounter over fitting dropout works more effectively than regularization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T08:06:51.863Z
Answers.LastActivityDate: 2020-05-14T08:06:51.863Z
Answers.UserDisplayName: Gerry P
Body: <p>I am training a multilayer neural network with 146 samples (97 for the training set, 20 for the validation set, and 29 for the testing set). I am using:</p>
  <ul>
  <li>automatic differentiation,</li>
  <li>SGD method,</li>
  <li>fixed learning rate + momentum term,</li>
  <li>logistic function,</li>
  <li>quadratic cost function,</li>
  <li>L1 and L2 regularization technique,</li>
  <li>adding some artificial noise 3%.</li>
  </ul>
  <p>When I used the L1 or L2 regularization technique, my problem (overfitting problem) got worst.</p>
  <p>I tried different values for lambdas (the penalty parameter 0.0001, 0.001, 0.01, 0.1, 1.0 and 5.0). After 0.1, I just killed my ANN. The best result that I took was using 0.001 (but it is worst comparing the one that I didn't use the regularization technique).</p>
  <p>The graph represents the error functions for different penalty parameters and also a case without using L1.</p>
  <p><a href="https://i.sstatic.net/NwBl2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NwBl2.png" alt="enter image description here" /></a></p>
  <p>and the accuracy</p>
  <p><a href="https://i.sstatic.net/MTgRD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MTgRD.png" alt="enter image description here" /></a></p>
  <p>What can be?</p>
  
Tags: <neural-networks><deep-learning><training><overfitting><regularization>
Title: Why did the L1/L2 regularization technique not improve my accuracy?
UserDisplayName: LVoltz
ThreadText: # Why did the L1/L2 regularization technique not improve my accuracy?
  
  (Asked by: username_0 on 2018-10-24)
  
  I am training a multilayer neural network with 146 samples (97 for the training set, 20 for the validation set, and 29 for the testing set). I am using:
  
  * automatic differentiation,
  * SGD method,
  * fixed learning rate + momentum term,
  * logistic function,
  * quadratic cost function,
  * L1 and L2 regularization technique,
  * adding some artificial noise 3%.
  
  When I used the L1 or L2 regularization technique, my problem (overfitting problem) got worst.
  
  I tried different values for lambdas (the penalty parameter 0.0001, 0.001, 0.01, 0.1, 1.0 and 5.0). After 0.1, I just killed my ANN. The best result that I took was using 0.001 (but it is worst comparing the one that I didn't use the regularization technique).
  
  The graph represents the error functions for different penalty parameters and also a case without using L1.
  
  [![enter image description here](https://i.sstatic.net/NwBl2.png)](https://i.sstatic.net/NwBl2.png)
  
  and the accuracy
  
  [![enter image description here](https://i.sstatic.net/MTgRD.png)](https://i.sstatic.net/MTgRD.png)
  
  What can be?
  
  **Tags:** neural-networks, deep-learning, training, overfitting, regularization
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-07-20. Score: 2
  
  You have a small dataset. Should you even be using neural nets? Have you done any diagnostics to see if you even have enough data? Are you using the right metric? Accuracy is not always the correct metric. Which weights are you retaining? You will overfit if you save the weights that produce the lowest training error. Save the weights that produced the lowest validation error. L1, L2, and dropout are all great. So many things not described in the problem...
  
  <http://www.ultravioletanalytics.com/blog/kaggle-titanic-competition-part-ix-bias-variance-and-learning-curves>
  
  I'm wondering why you're not trying interpretable models to see if the resulting weights for the features make sense. Also if your comparing all those models and parameters, set your random initial starting point to be the same by setting the seed. I also hope you are using the same training set for each model.
  
  You probably need more data...
  
  
  ---
  
  ### Answer by username_2 on 2020-05-14. Score: 0
  
  Your network without regularization does not appear to be over fitting but rather it appears to be converging to a minima. I am actually a bit surprised it is doing as well as it is given that your data set is small. So You don't need regularization. If you want to improve the accuracy you might try using an adjustable learning rate. The Keras call back ReduceLROnPlateau can be used for this. Documentation is [here](https://keras.io/api/callbacks/reduce_lr_on_plateau/). Also use the callback ModelCheckpoint to save the model with the lowest validation loss. Documentation is [here](https://keras.io/api/callbacks/model_checkpoint/). It would help a lot if you posted your model code. I have found if you do encounter over fitting dropout works more effectively than regularization.
  
  
  ---
  
  