Answers.Body: <p>One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is reasonable.</p>
  <p>Once you have a model working you can apply regularization if you think it will improve performance by reducing overfitting of the training data. You can check this by looking at the learning curves or compring the error on the validation and test sets.</p>
  <p>In my experiments I've used the L1 and L2 regularizers along with dropout. These can all be mixed together in fact using both L1 and L2 at the same time is called the ElasticNet.</p>
  <p>I tend to apply the regularizers on the <code>kernel_regularizer</code> because this affects the weights for the inputs. Basically feature selection.</p>
  <p>The value for the L1 and L2 can start with the default (for tensorflow) of 0.01 and change it as you see fit or read what other research papers have done.</p>
  <p>Dropout can start at 0.1 then increment it until there is no performance gain. This is basically a percentage so 0.1 would remove about 10% of your nodes.</p>
  <p>Finding the best regularizer is the same as any other hyperparameter optimization which is mostly trial and error.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-04T20:39:24.043Z
Answers.LastActivityDate: 2021-02-04T20:39:24.043Z
Answers.UserDisplayName: Marcus
Answers.Body: <p>If what is mentioned above, that is probably in the context of lstm networks. I would suggest using the keras tuner bayesian optimizer and making the l1 or l2 number a parameter of the kernel space. This way you find the optimal values, and its a great way to hypertune. Just keep in mind, the greater the range of parameters, or kernel if i am not wrong, the higher computer power you need.</p>
  <pre><code>from tensorflow import keras
  import keras_tuner as kt
  
  def model1(hp):
    model=Sequential()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                                dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                                recurrent_dropout=hp.Float('redroput',min_value=0.05, max_value=0.99, step=0.05),
                                activation='relu',
                                return_sequences=True,
                                input_shape=(30,1)))
    Attention()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                                dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                                activation='relu',return_sequences=True))
    Attention()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20), activation='relu'))
    model.add(keras.layers.Dense(1))
    
    model.compile(loss='mean_squared_error',optimizer=tf.keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4, 1e-7, 1e-10])))
    return model
  
  bayesian_opt_tuner = kt.BayesianOptimization(
      model1,
      objective='val_loss',
      max_trials=200,
      executions_per_trial=1,
      project_name='timeseries_bayes_opt_POC',
      overwrite=True,)
  
  xval=X_test
  bayesian_opt_tuner.search(x=X_train ,y=X_train, 
               epochs=300,
               #validation_data=(xval ,xval),
               validation_split=0.95,
               validation_steps=30,  
               steps_per_epoch=30,
               callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                patience=4,
                                verbose=1,
                                restore_best_weights=True),
                          tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', 
                                     factor=0.1, 
                                     patience= 2, 
                                     verbose=1, 
                                     min_delta=1e-5, 
                                     mode='min')]
               )
  This is where the magic happens. Something I composed myself. If interested holla 
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-11T23:53:55.297Z
Answers.LastActivityDate: 2021-09-11T23:53:55.297Z
Answers.UserDisplayName: aimonk
Answers.Body: <p>Regularization is trying to discourage complex information from being learned so we want to eliminate the model from actually learning to memorize the training data. We don't want to learn like very specific pinpoints of the training data that don't generalize well to test data.</p>
  <p>Dropout, the idea of drop out is that during training we randomly set some of the activations of the hidden neurons to zero with some probability say 0.5. This idea is extremely powerful because it allows the network to lower its capacity, it also makes it such that the network can't build these memorization channels through the network where it tries to just remember the data because on every iteration 50% of that data is going to be wiped out so it's going to be forced to not only generalize better but it's going to be forced to have multiple channels through the network and build a more robust representation of its prediction. We repeat this on every iteration so on the first iteration we dropped out one 50% of the nodes on the next iteration we can drop out a different randomly sampled 50% which may include some of the previously sampled nodes as well and this will allow the network to generalize better to new test data.</p>
  <p>Early stopping, when the network is improving its performance during training there comes a point where the training data starts to diverge from the testing data, at some point the network is going to start to do better on its training data than its testing data, what this means is basically that the network is starting to memorize some of the training data and that's what you don't want so what we can do is we can identify this inflection point where the test data starts to increase and diverge from the training data so we can stop the network early and make sure that our test accuracy is as minimum as possible.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-27T16:20:13.833Z
Answers.Text: The question was not 1. "What is regularization?" or 2. "How does dropout work" Instead, the questions were 1. "And how are these five parameters used?", 2. "For example, which parameters are most frequently used and what kind of value should be input?" (I believe this second question was the **main** question). Of course, the other question was assumed to be in the context of LSTMs and not in general. Your question only addresses "how to usually reduce over-fitting in general", but that wasn't really the question. As far as I understand, the question is clearly in the context of LSTMs.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-15T17:21:10.600Z
Answers.Text: How to apply these methods in LSTM?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-27T14:22:00.073Z
Answers.LastActivityDate: 2021-01-27T14:22:00.073Z
Answers.UserDisplayName: Faran Khalid
Body: <p>There are five parameters from an LSTM layer for regularization if I am correct.</p>
  <p>To deal with overfitting, I would start with</p>
  <ol>
  <li>reducing the layers</li>
  <li>reducing the hidden units</li>
  <li>Applying dropout or regularizers.</li>
  </ol>
  <p>There are <code>kernel_regularizer</code>, <code>recurrent_regularizer</code>, <code>bias_regularizer</code>, <code>activity_regularizer</code>, <code>dropout</code> and <code>recurrent_dropout</code>.</p>
  <p>They have their definitions on the Keras's website, but can anyone share more experiences on how to reduce overfitting?</p>
  <p>And how are these five parameters used? For example, which parameters are most frequently used and what kind of value should be input? ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-27T20:54:18.520Z
Comments.Text: This pre-print [Tikhonov Regularization for Long Short-Term Memory Networks](https://arxiv.org/pdf/1708.02979.pdf) could be useful: you may be already able to implement this in Keras. This paper [Recurrent Neural Network Regularization](https://arxiv.org/pdf/1409.2329v5.pdf) says that dropout does not work well in LSTMs and they suggest how to apply dropout to LSTMs so that it is effective. There's also this paper [Noisin: Unbiased Regularization for Recurrent Neural Networks](http://proceedings.mlr.press/v80/dieng18a/dieng18a.pdf).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-03T09:57:29.103Z
Comments.Text: In addition to that, you have [here](https://stackoverflow.com/q/48714407/3924118) a very similar post and [this article](https://machinelearningmastery.com/use-weight-regularization-lstm-networks-time-series-forecasting/) could be useful too. [Here](https://arxiv.org/pdf/1603.05118.pdf) is apparently the paper that introduced "recurrent dropout". There's also [this paper](https://arxiv.org/pdf/1606.01305.pdf) that could be useful.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-03T10:00:35.673Z
Comments.Text: You can probably find more related/relevant papers on the web. If I had more time, I would go through them and provide a formal answer, but not now. Feel free to read them and provide a formal answer to your own question!
Comments.UserDisplayName: nbro
Tags: <recurrent-neural-networks><long-short-term-memory><overfitting><regularization><dropout>
Title: How should we regularize an LSTM model?
UserDisplayName: Leo
ThreadText: # How should we regularize an LSTM model?
  
  (Asked by: username_0 on 2021-01-25)
  
  There are five parameters from an LSTM layer for regularization if I am correct.
  
  To deal with overfitting, I would start with
  
  1. reducing the layers
  2. reducing the hidden units
  3. Applying dropout or regularizers.
  
  There are `kernel_regularizer`, `recurrent_regularizer`, `bias_regularizer`, `activity_regularizer`, `dropout` and `recurrent_dropout`.
  
  They have their definitions on the Keras's website, but can anyone share more experiences on how to reduce overfitting?
  
  And how are these five parameters used? For example, which parameters are most frequently used and what kind of value should be input? ?
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, overfitting, regularization, dropout
  
  > **Comments:**
  > **username_1**: This pre-print [Tikhonov Regularization for Long Short-Term Memory Networks](https://arxiv.org/pdf/1708.02979.pdf) could be useful: you may be already able to implement this in Keras. This paper [Recurrent Neural Network Regularization](https://arxiv.org/pdf/1409.2329v5.pdf) says that dropout does not work well in LSTMs and they suggest how to apply dropout to LSTMs so that it is effective. There's also this paper [Noisin: Unbiased Regularization for Recurrent Neural Networks](http://proceedings.mlr.press/v80/dieng18a/dieng18a.pdf).  
  > **username_1**: In addition to that, you have [here](https://stackoverflow.com/q/48714407/3924118) a very similar post and [this article](https://machinelearningmastery.com/use-weight-regularization-lstm-networks-time-series-forecasting/) could be useful too. [Here](https://arxiv.org/pdf/1603.05118.pdf) is apparently the paper that introduced "recurrent dropout". There's also [this paper](https://arxiv.org/pdf/1606.01305.pdf) that could be useful.  
  > **username_1**: You can probably find more related/relevant papers on the web. If I had more time, I would go through them and provide a formal answer, but not now. Feel free to read them and provide a formal answer to your own question!  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2021-02-04. Score: 2
  
  One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is reasonable.
  
  Once you have a model working you can apply regularization if you think it will improve performance by reducing overfitting of the training data. You can check this by looking at the learning curves or compring the error on the validation and test sets.
  
  In my experiments I've used the L1 and L2 regularizers along with dropout. These can all be mixed together in fact using both L1 and L2 at the same time is called the ElasticNet.
  
  I tend to apply the regularizers on the `kernel_regularizer` because this affects the weights for the inputs. Basically feature selection.
  
  The value for the L1 and L2 can start with the default (for tensorflow) of 0.01 and change it as you see fit or read what other research papers have done.
  
  Dropout can start at 0.1 then increment it until there is no performance gain. This is basically a percentage so 0.1 would remove about 10% of your nodes.
  
  Finding the best regularizer is the same as any other hyperparameter optimization which is mostly trial and error.
  
  
  ---
  
  ### Answer by username_3 on 2021-09-11. Score: 1
  
  If what is mentioned above, that is probably in the context of lstm networks. I would suggest using the keras tuner bayesian optimizer and making the l1 or l2 number a parameter of the kernel space. This way you find the optimal values, and its a great way to hypertune. Just keep in mind, the greater the range of parameters, or kernel if i am not wrong, the higher computer power you need.
  
  ```
  from tensorflow import keras
  import keras_tuner as kt
  
  def model1(hp):
    model=Sequential()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                                dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                                recurrent_dropout=hp.Float('redroput',min_value=0.05, max_value=0.99, step=0.05),
                                activation='relu',
                                return_sequences=True,
                                input_shape=(30,1)))
    Attention()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                                dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                                activation='relu',return_sequences=True))
    Attention()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20), activation='relu'))
    model.add(keras.layers.Dense(1))
    
    model.compile(loss='mean_squared_error',optimizer=tf.keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4, 1e-7, 1e-10])))
    return model
  
  bayesian_opt_tuner = kt.BayesianOptimization(
      model1,
      objective='val_loss',
      max_trials=200,
      executions_per_trial=1,
      project_name='timeseries_bayes_opt_POC',
      overwrite=True,)
  
  xval=X_test
  bayesian_opt_tuner.search(x=X_train ,y=X_train, 
               epochs=300,
               #validation_data=(xval ,xval),
               validation_split=0.95,
               validation_steps=30,  
               steps_per_epoch=30,
               callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                patience=4,
                                verbose=1,
                                restore_best_weights=True),
                          tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', 
                                     factor=0.1, 
                                     patience= 2, 
                                     verbose=1, 
                                     min_delta=1e-5, 
                                     mode='min')]
               )
  This is where the magic happens. Something I composed myself. If interested holla 
  
  ```
  
  
  ---
  
  ### Answer by username_4 on 2021-01-27. Score: -1
  
  Regularization is trying to discourage complex information from being learned so we want to eliminate the model from actually learning to memorize the training data. We don't want to learn like very specific pinpoints of the training data that don't generalize well to test data.
  
  Dropout, the idea of drop out is that during training we randomly set some of the activations of the hidden neurons to zero with some probability say 0.5. This idea is extremely powerful because it allows the network to lower its capacity, it also makes it such that the network can't build these memorization channels through the network where it tries to just remember the data because on every iteration 50% of that data is going to be wiped out so it's going to be forced to not only generalize better but it's going to be forced to have multiple channels through the network and build a more robust representation of its prediction. We repeat this on every iteration so on the first iteration we dropped out one 50% of the nodes on the next iteration we can drop out a different randomly sampled 50% which may include some of the previously sampled nodes as well and this will allow the network to generalize better to new test data.
  
  Early stopping, when the network is improving its performance during training there comes a point where the training data starts to diverge from the testing data, at some point the network is going to start to do better on its training data than its testing data, what this means is basically that the network is starting to memorize some of the training data and that's what you don't want so what we can do is we can identify this inflection point where the test data starts to increase and diverge from the training data so we can stop the network early and make sure that our test accuracy is as minimum as possible.
  
  > **Comments:**
  > **username_1**: The question was not 1. "What is regularization?" or 2. "How does dropout work" Instead, the questions were 1. "And how are these five parameters used?", 2. "For example, which parameters are most frequently used and what kind of value should be input?" (I believe this second question was the **main** question). Of course, the other question was assumed to be in the context of LSTMs and not in general. Your question only addresses "how to usually reduce over-fitting in general", but that wasn't really the question. As far as I understand, the question is clearly in the context of LSTMs.  
  > **username_5**: How to apply these methods in LSTM?  
  
  ---
  
  