Answers.Body: <p>Couple reccomendations:<br>
  1) I dont think your overfitting, your test loss is not ever increasing and is staying reasonbly proportional to train loss -- This may indicate that whatever loss your using is not a good indicator of the metric of interest (in this case, it seems you want that to be accuracy, but data is imbalnced so maybe look at avg precision?)<br>
  2) Use a lr scheduler, or something like a ReduceLROnPlateau callback to reduce the lr once the loss converges<br>
  3) Adding more sources of the underrepresented class is valid but i reccomend just using class weighting. Effectively its the same (as an expectation) and will save you train time.  </p>
  
  <p>Good Luck!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-16T19:08:34.747Z
Answers.Text: I´m alreay using the `class_weight="balanced"` parameter inside the model.fit() call. Do you mean this with class weighting? And do you have an example with the avg precision for me?
Answers.UserDisplayName: JohnDizzle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-16T19:19:26.390Z
Answers.Text: @JohnDizzle if you pass class_weight a string itll just do nothing with it, you need to pass it a dict. Also if you balancing the classes on the data side, theres no reason to do it on the model side (i reccomend to just do it on the model side, no need to replicate data)
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-16T17:09:11.137Z
Answers.Text: Try adding dropout layers
Answers.UserDisplayName: Simbarashe Timothy Motsi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-16T19:02:09.727Z
Answers.LastActivityDate: 2019-05-16T19:02:09.727Z
Answers.UserDisplayName: mshlis
Body: <p>So I´m currently implementing my first neural network using GRUs as a model and Keras as an implementation since it´s pretty highlevel.
  My problem is about the classification of 8 hour long timeseries with 11 different events with 1 second timesteps or to be more specific: sleep recordings.<br>
  Since the GRU can´t handle the whole timeseries at once, I split it in 500 timepoint pieces with 50 seconds (10%) overlap.  Also since I don´t have much data, after the training/test split I´m oversampling the train data by duplicating the underrepresented classes up to 10 times. So at the end I get a set of ca. 14.000 training snippets and ca 1.800 to test the model. That to the data.</p>
  
  <p>Next thing is the model, represented with the Keras Code:</p>
  
  <pre><code>verbose, epochs, batch_size = 2, 50, 1200  # 1200 batch_size was the maximum the GPU can handle
      n_timesteps, n_features, n_outputs = trainX.shape[1], trainX.shape[2], trainy.shape[1]
  
      model = Sequential()
      model.add(GRU(128, input_shape=(n_timesteps, n_features), return_sequences=True, dropout=0.7,
                    kernel_regularizer=regularizers.l2(0.01), activation="relu"))
      model.add(
          GRU(64, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
              activation="relu"))
      model.add(
          GRU(32, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
              activation="relu"))
      model.add(
          GRU(16, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
              activation="relu"))
      model.add(TimeDistributed(Dense(units=16, activation='relu')))
      model.add(Flatten())
      model.add(Dense(n_outputs, activation='softmax'))
  
      # define custom optimizer with configurable learning rate
      sgd = optimizers.SGD(lr=0.1, momentum=0.9, decay=0.0, nesterov=False)
      model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
  </code></pre>
  
  <p>So as a short summary:<br>
  I am using 4 GRU layers with a "pyramid" shape, so the network has to be more specific towards the end. Also one fully connected layer as some kind of "adapter" and one output layer with the size of my features. I am using SGD as an optimizer.</p>
  
  <p>The results are always pretty bad, here are the loss and accuracy plots of an example run featuring 25 epochs:
  <img src="https://i.sstatic.net/gierx.png?s=250" alt="enter image description here">
  <a href="https://i.sstatic.net/IL81q.png?s=250" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IL81q.png?s=250" alt="enter image description here"></a></p>
  
  <p>Despite the huge dropout each stage, it still seems to be overfitting. Also as you can see, the accuracy of test and train is not changing after the 2nd epoch. The result is this sad looking confusion-matrix, showing only one class beeing detected by the network, which is not even the one with the highest amount of data in the dataset:  </p>
  
  <pre><code>[[231   0   0   0   0   0   0   0]
   [ 55   0   0   0   0   0   0   0]
   [647   0   0   0   0   0   0   0]
   [141   0   0   0   0   0   0   0]
   [444   0   0   0   0   0   0   0]
   [  0   0   0   0   0   0   0   0]
   [118   0   0   0   0   0   0   0]
   [ 74   0   0   0   0   0   0   0]]
  </code></pre>
  
  <p><strong>So what are possible approaches in order to:<br>
  a) fight the overfitting,<br>
  b) just one class beeing detected and<br>
  c) fix the stationary accuracy to be growing again?</strong></p>
  
  <p>Since I´m pretty new to neural networks I am thankful for your time and effort!</p>
  
Tags: <neural-networks><recurrent-neural-networks><keras><objective-functions><time-series>
Title: Train and Test Accuracy of GRU network not increasing after 2nd epoch
UserDisplayName: JohnDizzle
ThreadText: # Train and Test Accuracy of GRU network not increasing after 2nd epoch
  
  (Asked by: username_0 on 2019-05-16)
  
  So I´m currently implementing my first neural network using GRUs as a model and Keras as an implementation since it´s pretty highlevel.
  My problem is about the classification of 8 hour long timeseries with 11 different events with 1 second timesteps or to be more specific: sleep recordings.  
  Since the GRU can´t handle the whole timeseries at once, I split it in 500 timepoint pieces with 50 seconds (10%) overlap. Also since I don´t have much data, after the training/test split I´m oversampling the train data by duplicating the underrepresented classes up to 10 times. So at the end I get a set of ca. 14.000 training snippets and ca 1.800 to test the model. That to the data.
  
  Next thing is the model, represented with the Keras Code:
  
  ```
  verbose, epochs, batch_size = 2, 50, 1200  # 1200 batch_size was the maximum the GPU can handle
      n_timesteps, n_features, n_outputs = trainX.shape[1], trainX.shape[2], trainy.shape[1]
  
      model = Sequential()
      model.add(GRU(128, input_shape=(n_timesteps, n_features), return_sequences=True, dropout=0.7,
                    kernel_regularizer=regularizers.l2(0.01), activation="relu"))
      model.add(
          GRU(64, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
              activation="relu"))
      model.add(
          GRU(32, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
              activation="relu"))
      model.add(
          GRU(16, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
              activation="relu"))
      model.add(TimeDistributed(Dense(units=16, activation='relu')))
      model.add(Flatten())
      model.add(Dense(n_outputs, activation='softmax'))
  
      # define custom optimizer with configurable learning rate
      sgd = optimizers.SGD(lr=0.1, momentum=0.9, decay=0.0, nesterov=False)
      model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
  
  ```
  
  So as a short summary:  
  I am using 4 GRU layers with a "pyramid" shape, so the network has to be more specific towards the end. Also one fully connected layer as some kind of "adapter" and one output layer with the size of my features. I am using SGD as an optimizer.
  
  The results are always pretty bad, here are the loss and accuracy plots of an example run featuring 25 epochs:
  ![enter image description here](https://i.sstatic.net/gierx.png?s=250)
  [![enter image description here](https://i.sstatic.net/IL81q.png?s=250)](https://i.sstatic.net/IL81q.png?s=250)
  
  Despite the huge dropout each stage, it still seems to be overfitting. Also as you can see, the accuracy of test and train is not changing after the 2nd epoch. The result is this sad looking confusion-matrix, showing only one class beeing detected by the network, which is not even the one with the highest amount of data in the dataset:
  
  ```
  [[231   0   0   0   0   0   0   0]
   [ 55   0   0   0   0   0   0   0]
   [647   0   0   0   0   0   0   0]
   [141   0   0   0   0   0   0   0]
   [444   0   0   0   0   0   0   0]
   [  0   0   0   0   0   0   0   0]
   [118   0   0   0   0   0   0   0]
   [ 74   0   0   0   0   0   0   0]]
  
  ```
  
  **So what are possible approaches in order to:  
  a) fight the overfitting,  
  b) just one class beeing detected and  
  c) fix the stationary accuracy to be growing again?**
  
  Since I´m pretty new to neural networks I am thankful for your time and effort!
  
  **Tags:** neural-networks, recurrent-neural-networks, keras, objective-functions, time-series
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-16. Score: 1
  
  Couple reccomendations:  
  1) I dont think your overfitting, your test loss is not ever increasing and is staying reasonbly proportional to train loss -- This may indicate that whatever loss your using is not a good indicator of the metric of interest (in this case, it seems you want that to be accuracy, but data is imbalnced so maybe look at avg precision?)  
  2) Use a lr scheduler, or something like a ReduceLROnPlateau callback to reduce the lr once the loss converges  
  3) Adding more sources of the underrepresented class is valid but i reccomend just using class weighting. Effectively its the same (as an expectation) and will save you train time.
  
  Good Luck!
  
  > **Comments:**
  > **username_0**: I´m alreay using the `class_weight="balanced"` parameter inside the model.fit() call. Do you mean this with class weighting? And do you have an example with the avg precision for me?  
  > **username_1**: @username_0 if you pass class_weight a string itll just do nothing with it, you need to pass it a dict. Also if you balancing the classes on the data side, theres no reason to do it on the model side (i reccomend to just do it on the model side, no need to replicate data)  
  > **username_2**: Try adding dropout layers  
  
  ---
  
  