Answers.Body: <p>The solution for my problem was implementing Batch Renormalization: <code>BatchNormalization(renorm=True)</code>. In addition normalizing the inputs helped a lot improving the overall performance of the neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-13T17:11:55.607Z
Answers.LastActivityDate: 2021-02-13T17:11:55.607Z
Answers.UserDisplayName: raulfilipe127
Body: <p>The validation accuracy of my 1D CNN is stuck on 0.5 and that's because I'm always getting the same prediction out of a balanced data set. At the same time my training accuracy keeps increasing and the loss decreasing as intended.</p>
  <p>Strangely, if I do <code>model.evaluate()</code> on my training set (that has close to 1 accuracy in the last epoch), the accuracy will also be 0.5. How can the accuracy here differ so much from the training accuracy of the last epoch? I've also tried with a batch size of 1 for both training and evaluating and the problem persists.</p>
  <p>Well, I've been searching for different solutions for quite some time but still no luck. Possible problems I've already looked into:</p>
  <ol>
  <li>My data set is properly balanced and shuffled;</li>
  <li>My labels are correct;</li>
  <li>Tried adding fully connected layers;</li>
  <li>Tried adding/removing dropout from the fully connected layers;</li>
  <li>Tried the same architecture, but with the last layer with 1 neuron and sigmoid activation;</li>
  <li>Tried changing the learning rates (went down to 0.0001 but still the same problem).</li>
  </ol>
  <hr />
  <p><strong>Here's my code:</strong></p>
  <pre><code>import pathlib
  import numpy as np
  import ipynb.fs.defs.preprocessDataset as preprocessDataset
  import pickle
  import tensorflow as tf
  from tensorflow.keras.models import Sequential
  from tensorflow.keras import Input
  from tensorflow.keras.layers import Conv1D, BatchNormalization, Activation, MaxPooling1D, Flatten, Dropout, Dense
  from tensorflow.keras.optimizers import SGD
  
  main_folder = pathlib.Path.cwd().parent
  datasetsFolder=f'{main_folder}\\datasets'
  trainDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Train',datasetsFolder)
  testDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Test',datasetsFolder)
  
  X_train,Y_train,Names_train=trainDataset[0],trainDataset[1],trainDataset[2]
  X_test,Y_test,Names_test=testDataset[0],testDataset[1],testDataset[2]
  
  model = Sequential()
  
  model.add(Input(shape=X_train.shape[1:]))
  
  model.add(Conv1D(16, 61, strides=1, padding=&quot;same&quot;))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding=&quot;valid&quot;))
  
  model.add(Conv1D(32, 3, strides=1, padding=&quot;same&quot;))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding=&quot;valid&quot;))
  
  model.add(Conv1D(64, 3, strides=1, padding=&quot;same&quot;))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding=&quot;valid&quot;))
  
  model.add(Conv1D(64, 3, strides=1, padding=&quot;same&quot;))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding=&quot;valid&quot;))
  
  model.add(Conv1D(64, 3, strides=1, padding=&quot;same&quot;))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(Flatten())
  model.add(Dropout(0.5))
  
  model.add(Dense(200))
  model.add(Activation('relu'))
  
  model.add(Dense(2))
  model.add(Activation('softmax'))
  
  opt = SGD(learning_rate=0.01)
  
  model.compile(loss='binary_crossentropy',optimizer=opt,metrics=['accuracy'])
  
  model.summary()
  
  model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))
  
  model.evaluate(X_train,Y_train)
  </code></pre>
  <hr />
  <p><strong>Here's model.fit():</strong></p>
  <pre><code>model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))
  
  Epoch 1/10
  914/914 [==============================] - 277s 300ms/step - loss: 0.6405 - accuracy: 0.6543 - val_loss: 7.9835 - val_accuracy: 0.5000
  Epoch 2/10
  914/914 [==============================] - 270s 295ms/step - loss: 0.3997 - accuracy: 0.8204 - val_loss: 19.8981 - val_accuracy: 0.5000
  Epoch 3/10
  914/914 [==============================] - 273s 298ms/step - loss: 0.2976 - accuracy: 0.8730 - val_loss: 1.9558 - val_accuracy: 0.5002
  Epoch 4/10
  914/914 [==============================] - 278s 304ms/step - loss: 0.2897 - accuracy: 0.8776 - val_loss: 20.2678 - val_accuracy: 0.5000
  Epoch 5/10
  914/914 [==============================] - 277s 303ms/step - loss: 0.2459 - accuracy: 0.8991 - val_loss: 5.4945 - val_accuracy: 0.5000
  Epoch 6/10
  914/914 [==============================] - 268s 294ms/step - loss: 0.2008 - accuracy: 0.9181 - val_loss: 32.4579 - val_accuracy: 0.5000
  Epoch 7/10
  914/914 [==============================] - 271s 297ms/step - loss: 0.1695 - accuracy: 0.9317 - val_loss: 14.9538 - val_accuracy: 0.5000
  Epoch 8/10
  914/914 [==============================] - 276s 302ms/step - loss: 0.1423 - accuracy: 0.9452 - val_loss: 1.4420 - val_accuracy: 0.4988
  Epoch 9/10
  914/914 [==============================] - 266s 291ms/step - loss: 0.1261 - accuracy: 0.9497 - val_loss: 4.3830 - val_accuracy: 0.5005
  Epoch 10/10
  914/914 [==============================] - 272s 297ms/step - loss: 0.1142 - accuracy: 0.9548 - val_loss: 1.6054 - val_accuracy: 0.5009
  </code></pre>
  <p><strong>Here's model.evaluate():</strong></p>
  <pre><code>model.evaluate(X_train,Y_train)
  
  914/914 [==============================] - 35s 37ms/step - loss: 1.7588 - accuracy: 0.5009
  </code></pre>
  <p><strong>Here's model.summary():</strong></p>
  <pre><code>Model: &quot;sequential&quot;
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  conv1d (Conv1D)              (None, 4096, 16)          992       
  _________________________________________________________________
  batch_normalization (BatchNo (None, 4096, 16)          64        
  _________________________________________________________________
  activation (Activation)      (None, 4096, 16)          0         
  _________________________________________________________________
  max_pooling1d (MaxPooling1D) (None, 2048, 16)          0         
  _________________________________________________________________
  conv1d_1 (Conv1D)            (None, 2048, 32)          1568      
  _________________________________________________________________
  batch_normalization_1 (Batch (None, 2048, 32)          128       
  _________________________________________________________________
  activation_1 (Activation)    (None, 2048, 32)          0         
  _________________________________________________________________
  max_pooling1d_1 (MaxPooling1 (None, 1024, 32)          0         
  _________________________________________________________________
  conv1d_2 (Conv1D)            (None, 1024, 64)          6208      
  _________________________________________________________________
  batch_normalization_2 (Batch (None, 1024, 64)          256       
  _________________________________________________________________
  activation_2 (Activation)    (None, 1024, 64)          0         
  _________________________________________________________________
  max_pooling1d_2 (MaxPooling1 (None, 512, 64)           0         
  _________________________________________________________________
  conv1d_3 (Conv1D)            (None, 512, 64)           12352     
  _________________________________________________________________
  batch_normalization_3 (Batch (None, 512, 64)           256       
  _________________________________________________________________
  activation_3 (Activation)    (None, 512, 64)           0         
  _________________________________________________________________
  max_pooling1d_3 (MaxPooling1 (None, 256, 64)           0         
  _________________________________________________________________
  conv1d_4 (Conv1D)            (None, 256, 64)           12352     
  _________________________________________________________________
  batch_normalization_4 (Batch (None, 256, 64)           256       
  _________________________________________________________________
  activation_4 (Activation)    (None, 256, 64)           0         
  _________________________________________________________________
  flatten (Flatten)            (None, 16384)             0         
  _________________________________________________________________
  dropout (Dropout)            (None, 16384)             0         
  _________________________________________________________________
  dense (Dense)                (None, 200)               3277000   
  _________________________________________________________________
  activation_5 (Activation)    (None, 200)               0         
  _________________________________________________________________
  dense_1 (Dense)              (None, 2)                 402       
  _________________________________________________________________
  activation_6 (Activation)    (None, 2)                 0         
  =================================================================
  Total params: 3,311,834
  Trainable params: 3,311,354
  Non-trainable params: 480
  _________________________________________________________________
  </code></pre>
  
Tags: <deep-learning><convolutional-neural-networks><python><keras><1d-convolution>
Title: Keras 1D CNN always predicts the same result even if accuracy is high on training set
UserDisplayName: raulfilipe127
ThreadText: # Keras 1D CNN always predicts the same result even if accuracy is high on training set
  
  (Asked by: username_0 on 2021-01-17)
  
  The validation accuracy of my 1D CNN is stuck on 0.5 and that's because I'm always getting the same prediction out of a balanced data set. At the same time my training accuracy keeps increasing and the loss decreasing as intended.
  Strangely, if I do model.evaluate() on my training set (that has close to 1 accuracy in the last epoch), the accuracy will also be 0.5. How can the accuracy here differ so much from the training accuracy of the last epoch? I've also tried with a batch size of 1 for both training and evaluating and the problem persists.
  Well, I've been searching for different solutions for quite some time but still no luck. Possible problems I've already looked into:
  
  My data set is properly balanced and shuffled;
  My labels are correct;
  Tried adding fully connected layers;
  Tried adding/removing dropout from the fully connected layers;
  Tried the same architecture, but with the last layer with 1 neuron and sigmoid activation;
  Tried changing the learning rates (went down to 0.0001 but still the same problem).
  
  
  Here's my code:
  import pathlib
  import numpy as np
  import ipynb.fs.defs.preprocessDataset as preprocessDataset
  import pickle
  import tensorflow as tf
  from tensorflow.keras.models import Sequential
  from tensorflow.keras import Input
  from tensorflow.keras.layers import Conv1D, BatchNormalization, Activation, MaxPooling1D, Flatten, Dropout, Dense
  from tensorflow.keras.optimizers import SGD
  
  main_folder = pathlib.Path.cwd().parent
  datasetsFolder=f'{main_folder}\\datasets'
  trainDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Train',datasetsFolder)
  testDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Test',datasetsFolder)
  
  X_train,Y_train,Names_train=trainDataset[0],trainDataset[1],trainDataset[2]
  X_test,Y_test,Names_test=testDataset[0],testDataset[1],testDataset[2]
  
  model = Sequential()
  
  model.add(Input(shape=X_train.shape[1:]))
  
  model.add(Conv1D(16, 61, strides=1, padding="same"))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding="valid"))
  
  model.add(Conv1D(32, 3, strides=1, padding="same"))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding="valid"))
  
  model.add(Conv1D(64, 3, strides=1, padding="same"))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding="valid"))
  
  model.add(Conv1D(64, 3, strides=1, padding="same"))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling1D(2, strides=2, padding="valid"))
  
  model.add(Conv1D(64, 3, strides=1, padding="same"))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(Flatten())
  model.add(Dropout(0.5))
  
  model.add(Dense(200))
  model.add(Activation('relu'))
  
  model.add(Dense(2))
  model.add(Activation('softmax'))
  
  opt = SGD(learning_rate=0.01)
  
  model.compile(loss='binary_crossentropy',optimizer=opt,metrics=['accuracy'])
  
  model.summary()
  
  model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))
  
  model.evaluate(X_train,Y_train)
  
  
  Here's model.fit():
  model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))
  
  Epoch 1/10
  914/914 [==============================] - 277s 300ms/step - loss: 0.6405 - accuracy: 0.6543 - val_loss: 7.9835 - val_accuracy: 0.5000
  Epoch 2/10
  914/914 [==============================] - 270s 295ms/step - loss: 0.3997 - accuracy: 0.8204 - val_loss: 19.8981 - val_accuracy: 0.5000
  Epoch 3/10
  914/914 [==============================] - 273s 298ms/step - loss: 0.2976 - accuracy: 0.8730 - val_loss: 1.9558 - val_accuracy: 0.5002
  Epoch 4/10
  914/914 [==============================] - 278s 304ms/step - loss: 0.2897 - accuracy: 0.8776 - val_loss: 20.2678 - val_accuracy: 0.5000
  Epoch 5/10
  914/914 [==============================] - 277s 303ms/step - loss: 0.2459 - accuracy: 0.8991 - val_loss: 5.4945 - val_accuracy: 0.5000
  Epoch 6/10
  914/914 [==============================] - 268s 294ms/step - loss: 0.2008 - accuracy: 0.9181 - val_loss: 32.4579 - val_accuracy: 0.5000
  Epoch 7/10
  914/914 [==============================] - 271s 297ms/step - loss: 0.1695 - accuracy: 0.9317 - val_loss: 14.9538 - val_accuracy: 0.5000
  Epoch 8/10
  914/914 [==============================] - 276s 302ms/step - loss: 0.1423 - accuracy: 0.9452 - val_loss: 1.4420 - val_accuracy: 0.4988
  Epoch 9/10
  914/914 [==============================] - 266s 291ms/step - loss: 0.1261 - accuracy: 0.9497 - val_loss: 4.3830 - val_accuracy: 0.5005
  Epoch 10/10
  914/914 [==============================] - 272s 297ms/step - loss: 0.1142 - accuracy: 0.9548 - val_loss: 1.6054 - val_accuracy: 0.5009
  
  Here's model.evaluate():
  model.evaluate(X_train,Y_train)
  
  914/914 [==============================] - 35s 37ms/step - loss: 1.7588 - accuracy: 0.5009
  
  Here's model.summary():
  Model: "sequential"
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  conv1d (Conv1D)              (None, 4096, 16)          992       
  _________________________________________________________________
  batch_normalization (BatchNo (None, 4096, 16)          64        
  _________________________________________________________________
  activation (Activation)      (None, 4096, 16)          0         
  _________________________________________________________________
  max_pooling1d (MaxPooling1D) (None, 2048, 16)          0         
  _________________________________________________________________
  conv1d_1 (Conv1D)            (None, 2048, 32)          1568      
  _________________________________________________________________
  batch_normalization_1 (Batch (None, 2048, 32)          128       
  _________________________________________________________________
  activation_1 (Activation)    (None, 2048, 32)          0         
  _________________________________________________________________
  max_pooling1d_1 (MaxPooling1 (None, 1024, 32)          0         
  _________________________________________________________________
  conv1d_2 (Conv1D)            (None, 1024, 64)          6208      
  _________________________________________________________________
  batch_normalization_2 (Batch (None, 1024, 64)          256       
  _________________________________________________________________
  activation_2 (Activation)    (None, 1024, 64)          0         
  _________________________________________________________________
  max_pooling1d_2 (MaxPooling1 (None, 512, 64)           0         
  _________________________________________________________________
  conv1d_3 (Conv1D)            (None, 512, 64)           12352     
  _________________________________________________________________
  batch_normalization_3 (Batch (None, 512, 64)           256       
  _________________________________________________________________
  activation_3 (Activation)    (None, 512, 64)           0         
  _________________________________________________________________
  max_pooling1d_3 (MaxPooling1 (None, 256, 64)           0         
  _________________________________________________________________
  conv1d_4 (Conv1D)            (None, 256, 64)           12352     
  _________________________________________________________________
  batch_normalization_4 (Batch (None, 256, 64)           256       
  _________________________________________________________________
  activation_4 (Activation)    (None, 256, 64)           0         
  _________________________________________________________________
  flatten (Flatten)            (None, 16384)             0         
  _________________________________________________________________
  dropout (Dropout)            (None, 16384)             0         
  _________________________________________________________________
  dense (Dense)                (None, 200)               3277000   
  _________________________________________________________________
  activation_5 (Activation)    (None, 200)               0         
  _________________________________________________________________
  dense_1 (Dense)              (None, 2)                 402       
  _________________________________________________________________
  activation_6 (Activation)    (None, 2)                 0         
  =================================================================
  Total params: 3,311,834
  Trainable params: 3,311,354
  Non-trainable params: 480
  _________________________________________________________________
  
  **Tags:** deep-learning, convolutional-neural-networks, python, keras, 1d-convolution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-02-13. Score: 0 | Accepted answer
  
  The solution for my problem was implementing Batch Renormalization: `BatchNormalization(renorm=True)`. In addition normalizing the inputs helped a lot improving the overall performance of the neural network.
  
  
  ---
  
  