Answers.Body: <p>A couple of points:</p>
  
  <ol>
  <li><p>Have you firstly scaled your data, e.g. using <strong>MinMaxScaler</strong>? This could be one reason why your loss readings remain high.</p></li>
  <li><p>Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea.</p></li>
  </ol>
  
  <p>Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations.</p>
  
  <p><strong>Model without Dropout</strong></p>
  
  <pre><code># Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  </code></pre>
  
  <p>Over 20 epochs, the model achieves a validation loss of <strong>0.0267</strong> without Dropout.</p>
  
  <p><a href="https://i.sstatic.net/4HC1a.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4HC1a.png" alt="loss without dropout"></a></p>
  
  <p><strong>Model with Dropout</strong></p>
  
  <pre><code># Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dropout(0.5))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  </code></pre>
  
  <p>However, validation loss is slightly higher with Dropout at <strong>0.0428</strong>.</p>
  
  <p><a href="https://i.sstatic.net/0cje4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0cje4.png" alt="loss with dropout"></a></p>
  
  <ol start="3">
  <li>Make sure you have specified the loss function correctly. If you are forecasting a time series, then you are most likely working with interval data. Therefore, <strong>mean_squared_error</strong> is an appropriate loss function as one is trying to estimate the deviation between the predicted and actual values.</li>
  </ol>
  
  <p>As a counterexample, <strong>binary_crossentropy</strong> would not be suitable as the time series is not a classification set. However, misspecifying the loss function is a common error. Therefore, you also want to make sure you are using the appropriate loss function and then work from there.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T14:57:15.723Z
Answers.Text: 1. Yes, I scale data with StandardScaler
  2. This might be the answer, but why this happens? What's the remedy?
  3. Yes, I use MSE as a loss function
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T23:32:27.050Z
Answers.Text: Can you please include a plot of the time series itself (if that's possible)? It will be easier for me to answer if I know the structure of the series.
Answers.UserDisplayName: Michael Grogan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-07T14:01:08.470Z
Answers.Text: I added plots to the main question :)
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T22:04:46.213Z
Answers.Text: In your second graph, we can see a clear trend in the time series. From my experience, LSTMs don't tend to work very well with trend data. Here is an example: https://towardsdatascience.com/are-lstms-always-effective-at-time-series-analysis-a-case-study-of-oil-prices-97644f716a1c
Answers.UserDisplayName: Michael Grogan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-03T17:34:00.083Z
Answers.LastActivityDate: 2020-03-02T16:54:26.150Z
Answers.LastEditDate: 2020-03-02T16:54:26.150Z
Answers.UserDisplayName: Michael Grogan
Body: <p>I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows:</p>
  
  
  
  <pre><code>model = Sequential()
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  </code></pre>
  
  <p>As an effect my model is clearly overfitting:
  <a href="https://i.sstatic.net/CgP49.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CgP49.png" alt="enter image description here"></a></p>
  
  <p>So I decided to add dropout layers, first I added layers with 0.1, 0.3 and finally 0.5 rate:</p>
  
  
  
  <pre><code>model = Sequential()
  model.add(Dropout(0.5))
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  </code></pre>
  
  <p>However I think that it has no effect on the network learning process, even though 0.5 is quite large dropout rate:
  <a href="https://i.sstatic.net/Wz60u.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Wz60u.png" alt="enter image description here"></a></p>
  
  <p>Is this possible that dropout has little/no effect on a training process of LSTM  or maybe I do something wrong here?</p>
  
  <p>[EDIT] Adding plots of my TS, general and zoomed in view.
  <a href="https://i.sstatic.net/fK1vg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fK1vg.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/WDysd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WDysd.png" alt="enter image description here"></a></p>
  
  <p>I also want to add that the time of training increases just a bit (i.e. from 1540 to 1620 seconds) when I add the dropout layers.</p>
  
Tags: <neural-networks><keras><long-short-term-memory><dropout><regularization>
Title: Can dropout layers not influence LSTM training?
UserDisplayName: GKozinski
ThreadText: # Can dropout layers not influence LSTM training?
  
  (Asked by: username_0 on 2020-01-03)
  
  I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows:
  model = Sequential()
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  
  As an effect my model is clearly overfitting:
  
  So I decided to add dropout layers, first I added layers with 0.1, 0.3 and finally 0.5 rate:
  model = Sequential()
  model.add(Dropout(0.5))
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  
  However I think that it has no effect on the network learning process, even though 0.5 is quite large dropout rate:
  
  Is this possible that dropout has little/no effect on a training process of LSTM  or maybe I do something wrong here?
  [EDIT] Adding plots of my TS, general and zoomed in view.
  
  
  I also want to add that the time of training increases just a bit (i.e. from 1540 to 1620 seconds) when I add the dropout layers.
  
  **Tags:** neural-networks, keras, long-short-term-memory, dropout, regularization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-03. Score: 2
  
  A couple of points:
  
  Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high.
  Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea.
  
  Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations.
  Model without Dropout
  # Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  
  Over 20 epochs, the model achieves a validation loss of 0.0267 without Dropout.
  
  Model with Dropout
  # Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dropout(0.5))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  
  However, validation loss is slightly higher with Dropout at 0.0428.
  
  
  Make sure you have specified the loss function correctly. If you are forecasting a time series, then you are most likely working with interval data. Therefore, mean_squared_error is an appropriate loss function as one is trying to estimate the deviation between the predicted and actual values.
  
  As a counterexample, binary_crossentropy would not be suitable as the time series is not a classification set. However, misspecifying the loss function is a common error. Therefore, you also want to make sure you are using the appropriate loss function and then work from there.
  
  > **Comments:**
  > **username_0**: 1. Yes, I scale data with StandardScaler 2. This might be the answer, but why this happens? What's the remedy? 3. Yes, I use MSE as a loss function  
  > **username_1**: Can you please include a plot of the time series itself (if that's possible)? It will be easier for me to answer if I know the structure of the series.  
  > **username_0**: I added plots to the main question :)  
  > **username_1**: In your second graph, we can see a clear trend in the time series. From my experience, LSTMs don't tend to work very well with trend data. Here is an example: https://towardsdatascience.com/are-lstms-always-effective-at-time-series-analysis-a-case-study-of-oil-prices-97644f716a1c  
  
  ---
  
  