Answers.Body: <p>A couple of points:</p>
  
  <ol>
  <li><p>Have you firstly scaled your data, e.g. using <strong>MinMaxScaler</strong>? This could be one reason why your loss readings remain high.</p></li>
  <li><p>Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea.</p></li>
  </ol>
  
  <p>Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations.</p>
  
  <p><strong>Model without Dropout</strong></p>
  
  <pre><code># Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  </code></pre>
  
  <p>Over 20 epochs, the model achieves a validation loss of <strong>0.0267</strong> without Dropout.</p>
  
  <p><a href="https://i.sstatic.net/4HC1a.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4HC1a.png" alt="loss without dropout"></a></p>
  
  <p><strong>Model with Dropout</strong></p>
  
  <pre><code># Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dropout(0.5))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  </code></pre>
  
  <p>However, validation loss is slightly higher with Dropout at <strong>0.0428</strong>.</p>
  
  <p><a href="https://i.sstatic.net/0cje4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0cje4.png" alt="loss with dropout"></a></p>
  
  <ol start="3">
  <li>Make sure you have specified the loss function correctly. If you are forecasting a time series, then you are most likely working with interval data. Therefore, <strong>mean_squared_error</strong> is an appropriate loss function as one is trying to estimate the deviation between the predicted and actual values.</li>
  </ol>
  
  <p>As a counterexample, <strong>binary_crossentropy</strong> would not be suitable as the time series is not a classification set. However, misspecifying the loss function is a common error. Therefore, you also want to make sure you are using the appropriate loss function and then work from there.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T14:57:15.723Z
Answers.Text: 1. Yes, I scale data with StandardScaler
  2. This might be the answer, but why this happens? What's the remedy?
  3. Yes, I use MSE as a loss function
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T23:32:27.050Z
Answers.Text: Can you please include a plot of the time series itself (if that's possible)? It will be easier for me to answer if I know the structure of the series.
Answers.UserDisplayName: Michael Grogan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-07T14:01:08.470Z
Answers.Text: I added plots to the main question :)
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T22:04:46.213Z
Answers.Text: In your second graph, we can see a clear trend in the time series. From my experience, LSTMs don't tend to work very well with trend data. Here is an example: https://towardsdatascience.com/are-lstms-always-effective-at-time-series-analysis-a-case-study-of-oil-prices-97644f716a1c
Answers.UserDisplayName: Michael Grogan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-03T17:34:00.083Z
Answers.LastActivityDate: 2020-03-02T16:54:26.150Z
Answers.LastEditDate: 2020-03-02T16:54:26.150Z
Answers.UserDisplayName: Michael Grogan
Body: <p>I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows:</p>
  
  
  
  <pre><code>model = Sequential()
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  </code></pre>
  
  <p>As an effect my model is clearly overfitting:
  <a href="https://i.sstatic.net/CgP49.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CgP49.png" alt="enter image description here"></a></p>
  
  <p>So I decided to add dropout layers, first I added layers with 0.1, 0.3 and finally 0.5 rate:</p>
  
  
  
  <pre><code>model = Sequential()
  model.add(Dropout(0.5))
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  </code></pre>
  
  <p>However I think that it has no effect on the network learning process, even though 0.5 is quite large dropout rate:
  <a href="https://i.sstatic.net/Wz60u.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Wz60u.png" alt="enter image description here"></a></p>
  
  <p>Is this possible that dropout has little/no effect on a training process of LSTM  or maybe I do something wrong here?</p>
  
  <p>[EDIT] Adding plots of my TS, general and zoomed in view.
  <a href="https://i.sstatic.net/fK1vg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fK1vg.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/WDysd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WDysd.png" alt="enter image description here"></a></p>
  
  <p>I also want to add that the time of training increases just a bit (i.e. from 1540 to 1620 seconds) when I add the dropout layers.</p>
  
Tags: <neural-networks><keras><long-short-term-memory><dropout><regularization>
Title: Can dropout layers not influence LSTM training?
UserDisplayName: GKozinski
ThreadText: # Can dropout layers not influence LSTM training?
  
  (Asked by: username_0 on 2020-01-03)
  
  I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows:
  
  ```
  model = Sequential()
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  
  ```
  
  As an effect my model is clearly overfitting:
  [![enter image description here](https://i.sstatic.net/CgP49.png)](https://i.sstatic.net/CgP49.png)
  
  So I decided to add dropout layers, first I added layers with 0.1, 0.3 and finally 0.5 rate:
  
  ```
  model = Sequential()
  model.add(Dropout(0.5))
  model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128, return_sequences=True))
  model.add(Dropout(0.5))
  model.add(CuDNNLSTM(units=128))
  model.add(Dense(units=1))
  
  ```
  
  However I think that it has no effect on the network learning process, even though 0.5 is quite large dropout rate:
  [![enter image description here](https://i.sstatic.net/Wz60u.png)](https://i.sstatic.net/Wz60u.png)
  
  Is this possible that dropout has little/no effect on a training process of LSTM or maybe I do something wrong here?
  
  [EDIT] Adding plots of my TS, general and zoomed in view.
  [![enter image description here](https://i.sstatic.net/fK1vg.png)](https://i.sstatic.net/fK1vg.png)
  [![enter image description here](https://i.sstatic.net/WDysd.png)](https://i.sstatic.net/WDysd.png)
  
  I also want to add that the time of training increases just a bit (i.e. from 1540 to 1620 seconds) when I add the dropout layers.
  
  **Tags:** neural-networks, keras, long-short-term-memory, dropout, regularization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-03. Score: 2
  
  A couple of points:
  
  1. Have you firstly scaled your data, e.g. using **MinMaxScaler**? This could be one reason why your loss readings remain high.
  2. Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea.
  
  Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations.
  
  **Model without Dropout**
  
  ```
  # Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  
  ```
  
  Over 20 epochs, the model achieves a validation loss of **0.0267** without Dropout.
  
  [![loss without dropout](https://i.sstatic.net/4HC1a.png)](https://i.sstatic.net/4HC1a.png)
  
  **Model with Dropout**
  
  ```
  # Generate LSTM network
  model = tf.keras.Sequential()
  model.add(LSTM(4, input_shape=(1, previous)))
  model.add(Dropout(0.5))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)
  
  ```
  
  However, validation loss is slightly higher with Dropout at **0.0428**.
  
  [![loss with dropout](https://i.sstatic.net/0cje4.png)](https://i.sstatic.net/0cje4.png)
  
  3. Make sure you have specified the loss function correctly. If you are forecasting a time series, then you are most likely working with interval data. Therefore, **mean\_squared\_error** is an appropriate loss function as one is trying to estimate the deviation between the predicted and actual values.
  
  As a counterexample, **binary\_crossentropy** would not be suitable as the time series is not a classification set. However, misspecifying the loss function is a common error. Therefore, you also want to make sure you are using the appropriate loss function and then work from there.
  
  > **Comments:**
  > **username_0**: 1. Yes, I scale data with StandardScaler 2. This might be the answer, but why this happens? What's the remedy? 3. Yes, I use MSE as a loss function  
  > **username_1**: Can you please include a plot of the time series itself (if that's possible)? It will be easier for me to answer if I know the structure of the series.  
  > **username_0**: I added plots to the main question :)  
  > **username_1**: In your second graph, we can see a clear trend in the time series. From my experience, LSTMs don't tend to work very well with trend data. Here is an example: https://towardsdatascience.com/are-lstms-always-effective-at-time-series-analysis-a-case-study-of-oil-prices-97644f716a1c  
  
  ---
  
  