Answers.Body: <p>It looks like you're defining an optimizer in the line <code>sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)</code> and then not using it, since you compile the model with <code>optimizer='adam'</code>. So changing any of the sgd parameters, or even commenting out the line, changes nothing for your model.</p>
  <p>you'll need to change to <code>model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])</code> to use the <code>sgd</code> optimizer object in the model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-18T11:50:09.593Z
Answers.Text: Oh,  I didn't pay attention to that. Ok I will try it.
Answers.UserDisplayName: root
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-18T09:26:39.023Z
Answers.LastActivityDate: 2022-08-18T09:26:39.023Z
Answers.UserDisplayName: Kroshtan
Body: <p>I am trying to retrain ResNet-50 for iris flower classification in tensorflow (TensorFlow version: 2.3.0) using the following code</p>
  <pre><code>import tensorflow as tf
  import cv2, random
  from sklearn.model_selection import train_test_split
  from sklearn.metrics import classification_report
  from random import shuffle 
  from IPython.display import SVG
  import numpy as np # linear algebra
  import pandas as pd 
  import shutil
  import matplotlib.pyplot as plt
  %matplotlib inline 
  from IPython.display import Image, display
  from sklearn.model_selection import train_test_split
  import os
  print(os.listdir(&quot;./iris recognition/flowers&quot;))
  
  labels = os.listdir(&quot;./iris recognition/flowers&quot;)
  num_classes = len(set(labels))
  IMAGE_SIZE= 224
  
  
  # Create model
  model = tf.keras.Sequential()
  model.add(tf.keras.applications.ResNet50(include_top=False, weights='imagenet'))
  model.add(tf.keras.layers.GlobalAveragePooling2D())
  model.add(tf.keras.layers.Dropout(0.5))
  model.add(tf.keras.layers.Dense(num_classes, activation='softmax'))
  
  # Do not train first layer (ResNet) as it is already pre-trained
  model.layers[0].trainable = False
  
  # Compile model
  from tensorflow.python.keras import optimizers
  
  sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  model.summary()
  train_folder = './iris recognition/flowers'
  
  image_size = 224
  data_generator =  tf.keras.preprocessing.image.ImageDataGenerator(preprocessing_function=tf.keras.applications.resnet50.preprocess_input,
                                      horizontal_flip=True,
                                      width_shift_range=0.2,
                                      height_shift_range=0.2,
                                      validation_split=0.2)# set validation split
  
  train_generator = data_generator.flow_from_directory(
      train_folder,
      target_size=(image_size, image_size),
      batch_size=10,
      class_mode='categorical',
      subset='training'
      )
  validation_generator = data_generator.flow_from_directory(
      train_folder,
      target_size=(image_size, image_size),
      batch_size=10,
      class_mode='categorical',
      subset='validation'
      )
  
  NUM_EPOCHS = 70
  EARLY_STOP_PATIENCE = 5
  from tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint
  
  cb_early_stopper = EarlyStopping(monitor = 'val_loss', patience = EARLY_STOP_PATIENCE)
  cb_checkpointer = ModelCheckpoint(filepath = './working/best.hdf5',
                                    monitor = 'val_loss',
                                    save_best_only = True,
                                    mode = 'auto')
  import math
  
  fit_history = model.fit(
      train_generator,
      steps_per_epoch=10,
      validation_data=validation_generator,
  #     validation_steps=10,
      epochs=NUM_EPOCHS,
      callbacks=[cb_checkpointer, cb_early_stopper])
  model.load_weights(&quot;./working/best.hdf5&quot;)
  </code></pre>
  <p>I tried to change the number of epochs , steps_per_epoch and validation_steps however the model accuracy does not improved as shown in <a href="https://i.sstatic.net/OKz3o.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OKz3o.png" alt="figure1" /></a> <a href="https://i.sstatic.net/3qmaO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3qmaO.png" alt="figure2" /></a></p>
  <p>After trying several essais while changing batch size from 32, 64 and 128.For each essai I tried with and without using :</p>
  <ul>
  <li><p>steps_per_epoch=10.</p>
  </li>
  <li><p>sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9,
  nesterov = True)</p>
  </li>
  </ul>
  <p>The best accuracy I achieved is 72.29% in which the training was the most stable which was when using batch size=128 as shwon in <a href="https://i.sstatic.net/oNRL6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oNRL6.png" alt="figure3" /></a></p>
  <p>Any suggestions please?!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-15T17:16:55.693Z
Comments.Text: Have you tried decreasing the learning rate or increasing the batch size? Your model is fluctuating widely and the lr. or the batch size might be reasons for that.
Comments.UserDisplayName: Mariusmarten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-15T17:21:19.717Z
Comments.Text: I only tried to change the batch size form 24 to 10.
Comments.UserDisplayName: root
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-15T17:23:04.993Z
Comments.Text: You could try batch sizes of 32 or 64 (or even 128), this might stabilize the training already.
Comments.UserDisplayName: Mariusmarten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-16T10:09:42.543Z
Comments.Text: Actually, I tried as you said changing with these parameters as follow:  I tried changing the batch size from 32 ,64 and 128 **with sgd** (with and without using steps per epoch) and **without sgd**(with and without using steps per epoch) and the best accuracy I achieved is 72.29%
Comments.UserDisplayName: root
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-16T20:14:59.507Z
Comments.Text: Has training become more steady? Have you tried a smaller learning rate as well?
Comments.UserDisplayName: Mariusmarten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T19:29:51.890Z
Comments.Text: @Mariusmarten, I have edited the question with the last updates. Considering the learning rate which is in my code **sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)** I just comment this line and run my code.
Comments.UserDisplayName: root
Tags: <tensorflow><residual-networks><iris-dataset>
Title: Retraining ResNet-50 for iris flower classification
UserDisplayName: root
ThreadText: # Retraining ResNet-50 for iris flower classification
  
  (Asked by: username_0 on 2022-08-15)
  
  I am trying to retrain ResNet-50 for iris flower classification in tensorflow (TensorFlow version: 2.3.0) using the following code
  
  ```
  import tensorflow as tf
  import cv2, random
  from sklearn.model_selection import train_test_split
  from sklearn.metrics import classification_report
  from random import shuffle 
  from IPython.display import SVG
  import numpy as np # linear algebra
  import pandas as pd 
  import shutil
  import matplotlib.pyplot as plt
  %matplotlib inline 
  from IPython.display import Image, display
  from sklearn.model_selection import train_test_split
  import os
  print(os.listdir("./iris recognition/flowers"))
  
  labels = os.listdir("./iris recognition/flowers")
  num_classes = len(set(labels))
  IMAGE_SIZE= 224
  
  # Create model
  model = tf.keras.Sequential()
  model.add(tf.keras.applications.ResNet50(include_top=False, weights='imagenet'))
  model.add(tf.keras.layers.GlobalAveragePooling2D())
  model.add(tf.keras.layers.Dropout(0.5))
  model.add(tf.keras.layers.Dense(num_classes, activation='softmax'))
  
  # Do not train first layer (ResNet) as it is already pre-trained
  model.layers[0].trainable = False
  
  # Compile model
  from tensorflow.python.keras import optimizers
  
  sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  model.summary()
  train_folder = './iris recognition/flowers'
  
  image_size = 224
  data_generator =  tf.keras.preprocessing.image.ImageDataGenerator(preprocessing_function=tf.keras.applications.resnet50.preprocess_input,
                                      horizontal_flip=True,
                                      width_shift_range=0.2,
                                      height_shift_range=0.2,
                                      validation_split=0.2)# set validation split
  
  train_generator = data_generator.flow_from_directory(
      train_folder,
      target_size=(image_size, image_size),
      batch_size=10,
      class_mode='categorical',
      subset='training'
      )
  validation_generator = data_generator.flow_from_directory(
      train_folder,
      target_size=(image_size, image_size),
      batch_size=10,
      class_mode='categorical',
      subset='validation'
      )
  
  NUM_EPOCHS = 70
  EARLY_STOP_PATIENCE = 5
  from tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint
  
  cb_early_stopper = EarlyStopping(monitor = 'val_loss', patience = EARLY_STOP_PATIENCE)
  cb_checkpointer = ModelCheckpoint(filepath = './working/best.hdf5',
                                    monitor = 'val_loss',
                                    save_best_only = True,
                                    mode = 'auto')
  import math
  
  fit_history = model.fit(
      train_generator,
      steps_per_epoch=10,
      validation_data=validation_generator,
  #     validation_steps=10,
      epochs=NUM_EPOCHS,
      callbacks=[cb_checkpointer, cb_early_stopper])
  model.load_weights("./working/best.hdf5")
  
  ```
  
  I tried to change the number of epochs , steps\_per\_epoch and validation\_steps however the model accuracy does not improved as shown in [![figure1](https://i.sstatic.net/OKz3o.png)](https://i.sstatic.net/OKz3o.png) [![figure2](https://i.sstatic.net/3qmaO.png)](https://i.sstatic.net/3qmaO.png)
  
  After trying several essais while changing batch size from 32, 64 and 128.For each essai I tried with and without using :
  
  * steps\_per\_epoch=10.
  * sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9,
    nesterov = True)
  
  The best accuracy I achieved is 72.29% in which the training was the most stable which was when using batch size=128 as shwon in [![figure3](https://i.sstatic.net/oNRL6.png)](https://i.sstatic.net/oNRL6.png)
  
  Any suggestions please?!
  
  **Tags:** tensorflow, residual-networks, iris-dataset
  
  > **Comments:**
  > **username_1**: Have you tried decreasing the learning rate or increasing the batch size? Your model is fluctuating widely and the lr. or the batch size might be reasons for that.  
  > **username_0**: I only tried to change the batch size form 24 to 10.  
  > **username_1**: You could try batch sizes of 32 or 64 (or even 128), this might stabilize the training already.  
  > **username_0**: Actually, I tried as you said changing with these parameters as follow:  I tried changing the batch size from 32 ,64 and 128 **with sgd** (with and without using steps per epoch) and **without sgd**(with and without using steps per epoch) and the best accuracy I achieved is 72.29%  
  > **username_1**: Has training become more steady? Have you tried a smaller learning rate as well?  
  > **username_0**: @username_1, I have edited the question with the last updates. Considering the learning rate which is in my code **sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)** I just comment this line and run my code.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-08-18. Score: 1
  
  It looks like you're defining an optimizer in the line `sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)` and then not using it, since you compile the model with `optimizer='adam'`. So changing any of the sgd parameters, or even commenting out the line, changes nothing for your model.
  
  you'll need to change to `model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])` to use the `sgd` optimizer object in the model.
  
  > **Comments:**
  > **username_0**: Oh,  I didn't pay attention to that. Ok I will try it.  
  
  ---
  
  