Answers.Body: <p>What is the No Information Rate (NIR)? I.e. what are the percentages of positive and negative labels? Have you looked at the predictions of your model? If it's all 0's or all 1's then it probably learned nothing, other than predicting the majority class.</p>
  <p>When it comes to architectural choices and hyperparameters, especially if you start working with NNs, then Andrej Karpathy's blog post called <a href="http://karpathy.github.io/2019/04/25/recipe/" rel="nofollow noreferrer">A Recipe for Training Neural Networks</a> is a really good starting point. It gives a good reference on how to approach things in the beginning when you have not much intuition. Simply reducing the learning rate will not help much if your model is way too small. You may also find it useful to add <a href="https://en.wikipedia.org/wiki/Residual_neural_network" rel="nofollow noreferrer">ResNet</a>-like skip connections to improve performance for very deep models (i.e. many layer).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-11T15:21:51.703Z
Answers.LastActivityDate: 2021-10-11T15:21:51.703Z
Answers.UserDisplayName: Tinu
Answers.Body: <p>Try removing the dropout before the prediction layer. I couldn't find the paper or article I read about this (will update the post once I find it), just found a <a href="https://stats.stackexchange.com/questions/299292/dropout-makes-performance-worse/299305">Cross Validated post</a> which does not add much information. As you are</p>
  <p>If you are lowering the learning rate, you should also lower the <a href="https://stackoverflow.com/questions/53033556/how-should-the-learning-rate-change-as-the-batch-size-change/53046624#53046624">batch size accordingly</a>.</p>
  <p>As for Batch Normalization layers, they probably should be applied <a href="https://stackoverflow.com/a/47143624/9420000">after the convolutional layers</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-12T09:34:12.950Z
Answers.LastActivityDate: 2021-10-12T09:34:12.950Z
Answers.UserDisplayName: serali
Body: <p>I have used a 3D CNN architecture, for detecting the presence of a particular promoter (MGMT), by using FLAIR brain scans. (64 slices per patient). The output is supposed to be binary (0/1).</p>
  <p>I have gone through the pre-processing properly, and used stratification after splitting the &quot;train&quot; dataset into train and validation sets, (80-20 ratio). My model initialisation and training kernels look like this:</p>
  <pre><code>def get_model(width=128, height=128, depth=64):
  &quot;&quot;&quot;Build a 3D convolutional neural network model.&quot;&quot;&quot;
  
  inputs = keras.Input((width, height, depth, 1))
  
  x = layers.Conv3D(filters=64, kernel_size=3, activation=&quot;relu&quot;)(inputs)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.Conv3D(filters=64, kernel_size=3, activation=&quot;relu&quot;)(x)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.Conv3D(filters=128, kernel_size=3, activation=&quot;relu&quot;)(x)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.Conv3D(filters=256, kernel_size=3, activation=&quot;relu&quot;)(x)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.GlobalAveragePooling3D()(x)
  x = layers.Dense(units=512, activation=&quot;relu&quot;)(x)
  x = layers.Dropout(0.3)(x)
  
  outputs = layers.Dense(units=1, activation=&quot;sigmoid&quot;)(x)
  
  # Define the model.
  model = keras.Model(inputs, outputs, name=&quot;3dcnn&quot;)
  return model
  
  
  # Build model.
  model = get_model(width=128, height=128, depth=64)
  model.summary()
  </code></pre>
  <h1>Compile model:</h1>
  <pre><code>initial_learning_rate = 0.0001
  lr_schedule = keras.optimizers.schedules.ExponentialDecay(
      initial_learning_rate, decay_steps=100000, decay_rate=0.96, staircase=True
  )
  model.compile(
      loss=&quot;binary_crossentropy&quot;,
      optimizer=keras.optimizers.Adam(learning_rate=lr_schedule),
      metrics=[&quot;acc&quot;],
  )
  
  # Define callbacks.
  checkpoint_cb = keras.callbacks.ModelCheckpoint(
      &quot;Brain_3d_classification.h5&quot;, save_best_only=True,monitor = 'val_acc', 
                               mode = 'max', verbose = 1
  )
  early_stopping_cb = keras.callbacks.EarlyStopping(monitor=&quot;val_acc&quot;, patience=20,mode = 'max', verbose = 1,
                             restore_best_weights = True)
  
  # Train the model, doing validation at the end of each epoch
  epochs = 60
  model.fit(
      train_dataset,
      validation_data=valid_dataset,
      epochs=epochs,
      shuffle=True,
      verbose=2,
      callbacks=[checkpoint_cb, early_stopping_cb],
  )
  </code></pre>
  <p>This is my first time ever working with a 3D CNN, and I used this keras webpage for the format:<a href="https://keras.io/examples/vision/3D_image_classification/" rel="nofollow noreferrer">https://keras.io/examples/vision/3D_image_classification/</a></p>
  <p>The (max) validation accuracy in my case was about 54%. I tried reducing the initial learning rate , and for 0.00001 I got to a max of 66.7%. For learning rates of 0.00005, 0.00002, I got max accuracy of about 60 and 62%.</p>
  <p>Accuracy vs epoch plots for learning rates 0.0001, 0.00005,0.00002 and 0.00001:</p>
  <p><a href="https://i.sstatic.net/FYeqH.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FYeqH.jpg" alt="0.0001" /></a>
  <a href="https://i.sstatic.net/zsJg7.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zsJg7.jpg" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/qT4xZ.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qT4xZ.jpg" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/IV2GQ.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IV2GQ.jpg" alt="enter image description here" /></a></p>
  <p>It does seem like reducing the initial learning rate has a positive effect on accuracy, although the accuracy is still very low.</p>
  <p><strong>What other parameters can I tune to expect a better accuracy? And is it okay to just keep reducing the initial learning rate until we achieve a targeted accuracy?</strong></p>
  <p>I know this is a rather broad question, but I am quite confused as to how we should approach increasing the accuracy in the case of CNNs, (that too 3D), where there just seems to be a lot of stuff going on. Do I change something in my initialisations? Add more layers? Or change the parameters? Do I decrease or increase them? With so many things going on, I don't think trying every combination and just keep repeating the training process is an efficient idea...</p>
  <p>Full notebook (including pre-processing steps): <a href="https://www.kaggle.com/shivamee682003/3d-image-preprocessing-17cd03/edit" rel="nofollow noreferrer">https://www.kaggle.com/shivamee682003/3d-image-preprocessing-17cd03/edit</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-12T09:34:37.110Z
Comments.Text: link you provide to the notebook is not working.
Comments.UserDisplayName: serali
Tags: <neural-networks><convolutional-neural-networks><keras><3d-convolution>
Title: Improving validation losses and accuracy for 3D CNN
UserDisplayName: satan 29
ThreadText: # Improving validation losses and accuracy for 3D CNN
  
  (Asked by: username_0 on 2021-10-11)
  
  I have used a 3D CNN architecture, for detecting the presence of a particular promoter (MGMT), by using FLAIR brain scans. (64 slices per patient). The output is supposed to be binary (0/1).
  I have gone through the pre-processing properly, and used stratification after splitting the "train" dataset into train and validation sets, (80-20 ratio). My model initialisation and training kernels look like this:
  def get_model(width=128, height=128, depth=64):
  """Build a 3D convolutional neural network model."""
  
  inputs = keras.Input((width, height, depth, 1))
  
  x = layers.Conv3D(filters=64, kernel_size=3, activation="relu")(inputs)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.Conv3D(filters=64, kernel_size=3, activation="relu")(x)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.Conv3D(filters=128, kernel_size=3, activation="relu")(x)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.Conv3D(filters=256, kernel_size=3, activation="relu")(x)
  x = layers.MaxPool3D(pool_size=2)(x)
  x = layers.BatchNormalization()(x)
  
  x = layers.GlobalAveragePooling3D()(x)
  x = layers.Dense(units=512, activation="relu")(x)
  x = layers.Dropout(0.3)(x)
  
  outputs = layers.Dense(units=1, activation="sigmoid")(x)
  
  # Define the model.
  model = keras.Model(inputs, outputs, name="3dcnn")
  return model
  
  
  # Build model.
  model = get_model(width=128, height=128, depth=64)
  model.summary()
  
  Compile model:
  initial_learning_rate = 0.0001
  lr_schedule = keras.optimizers.schedules.ExponentialDecay(
      initial_learning_rate, decay_steps=100000, decay_rate=0.96, staircase=True
  )
  model.compile(
      loss="binary_crossentropy",
      optimizer=keras.optimizers.Adam(learning_rate=lr_schedule),
      metrics=["acc"],
  )
  
  # Define callbacks.
  checkpoint_cb = keras.callbacks.ModelCheckpoint(
      "Brain_3d_classification.h5", save_best_only=True,monitor = 'val_acc', 
                               mode = 'max', verbose = 1
  )
  early_stopping_cb = keras.callbacks.EarlyStopping(monitor="val_acc", patience=20,mode = 'max', verbose = 1,
                             restore_best_weights = True)
  
  # Train the model, doing validation at the end of each epoch
  epochs = 60
  model.fit(
      train_dataset,
      validation_data=valid_dataset,
      epochs=epochs,
      shuffle=True,
      verbose=2,
      callbacks=[checkpoint_cb, early_stopping_cb],
  )
  
  This is my first time ever working with a 3D CNN, and I used this keras webpage for the format:https://keras.io/examples/vision/3D_image_classification/
  The (max) validation accuracy in my case was about 54%. I tried reducing the initial learning rate , and for 0.00001 I got to a max of 66.7%. For learning rates of 0.00005, 0.00002, I got max accuracy of about 60 and 62%.
  Accuracy vs epoch plots for learning rates 0.0001, 0.00005,0.00002 and 0.00001:
  
  
  
  
  It does seem like reducing the initial learning rate has a positive effect on accuracy, although the accuracy is still very low.
  What other parameters can I tune to expect a better accuracy? And is it okay to just keep reducing the initial learning rate until we achieve a targeted accuracy?
  I know this is a rather broad question, but I am quite confused as to how we should approach increasing the accuracy in the case of CNNs, (that too 3D), where there just seems to be a lot of stuff going on. Do I change something in my initialisations? Add more layers? Or change the parameters? Do I decrease or increase them? With so many things going on, I don't think trying every combination and just keep repeating the training process is an efficient idea...
  Full notebook (including pre-processing steps): https://www.kaggle.com/shivamee682003/3d-image-preprocessing-17cd03/edit
  
  **Tags:** neural-networks, convolutional-neural-networks, keras, 3d-convolution
  > **Comments:**
  > **username_1**: link you provide to the notebook is not working.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-10-11. Score: 1
  
  What is the No Information Rate (NIR)? I.e. what are the percentages of positive and negative labels? Have you looked at the predictions of your model? If it's all 0's or all 1's then it probably learned nothing, other than predicting the majority class.
  
  When it comes to architectural choices and hyperparameters, especially if you start working with NNs, then Andrej Karpathy's blog post called [A Recipe for Training Neural Networks](http://karpathy.github.io/2019/04/25/recipe/) is a really good starting point. It gives a good reference on how to approach things in the beginning when you have not much intuition. Simply reducing the learning rate will not help much if your model is way too small. You may also find it useful to add [ResNet](https://en.wikipedia.org/wiki/Residual_neural_network)-like skip connections to improve performance for very deep models (i.e. many layer).
  
  
  ---
  
  ### Answer by username_1 on 2021-10-12. Score: 1
  
  Try removing the dropout before the prediction layer. I couldn't find the paper or article I read about this (will update the post once I find it), just found a [Cross Validated post](https://stats.stackexchange.com/questions/299292/dropout-makes-performance-worse/299305) which does not add much information. As you are
  
  If you are lowering the learning rate, you should also lower the [batch size accordingly](https://stackoverflow.com/questions/53033556/how-should-the-learning-rate-change-as-the-batch-size-change/53046624#53046624).
  
  As for Batch Normalization layers, they probably should be applied [after the convolutional layers](https://stackoverflow.com/a/47143624/9420000).
  
  
  ---
  
  