Answers.Body: <p>I found the issue with a tip from stackoverflow!</p>
  <p>The problem is the normalization in the inference script. It is supposed to be <code>input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr)</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-22T09:23:04.273Z
Answers.LastActivityDate: 2022-08-22T09:23:04.273Z
Answers.UserDisplayName: Sohrab Tawana
Body: <p>I use <code>tensorflow keras</code> to build a neural network that classifies images of covid-19 rapid tests into three classes (<a href="https://i.sstatic.net/M53bS.jpg" rel="nofollow noreferrer">Negative</a>, <a href="https://i.sstatic.net/uqiRV.jpg" rel="nofollow noreferrer">Positive</a>, <a href="https://i.sstatic.net/SBLrd.jpg" rel="nofollow noreferrer">Empty</a>).</p>
  <p>During training the tensorboard logs denote a validation accuracy of around 90%. But when I test the network after being trained with the same images it was trained on, the classification performance is way worse (~60%). I observed the same behavior when I trained the network with different images (see section <code>What I have tried</code>).</p>
  <p>During training the images are preprocessed to grayscale and resized before being fed into the model. The batch size is 16.</p>
  <pre><code>image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  </code></pre>
  <p>To augment the sparse data that I have (~450 images) I am using the <code>keras.preprocessing.image.ImageDataGenerator</code> and its parameters are: <code>width_shift_range=0.1</code>, <code>brightness_range=[0.9, 1.1]</code>, <code>height_shift_range=0.1</code>, <code>zoom_range=0.2</code>, <code>horizontal_flip=True</code>, <code>rotation_range=10</code>, <code>shear_range=0.2</code>, <code>fill_mode=&quot;nearest&quot;</code>, <code>samplewise_center=True</code>, <code>samplewise_std_normalization=True</code></p>
  <p>Edit: The images are split into 90/10 training/validation sets.</p>
  <p><code>(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.1, random_state=24)</code></p>
  <p>I am converting the model to tflite because we need it for mobile platforms. I am using this code snippet:</p>
  <pre><code>model = tf.keras.models.load_model(model_path)
  
  converter = tf.lite.TFLiteConverter.from_keras_model(model) # path to the SavedModel directory
  # converter.optimizations = [tf.lite.Optimize.DEFAULT] # optimizations
  tflite_model = converter.convert()
  
  # Save the model.
  with open('rapid_test_strip_cleaned_model.tflite', 'wb') as f:
    f.write(tflite_model)
  </code></pre>
  <p><strong>What I have tried:</strong></p>
  <ul>
  <li>crop the images to the <a href="https://i.sstatic.net/CCKtS.jpg" rel="nofollow noreferrer">strip</a> of the casette, train and test the network again</li>
  <li>check in the testing (inference) script if the labels are correct</li>
  <li>check if the images are converted to grayscale and resized correctly before being fed into the network during testing</li>
  <li>test the model before converting it to tflite, using <code>tensorflow.keras.models</code></li>
  </ul>
  <p><strong>Model:</strong></p>
  <pre><code>img_width, img_height = (256, 256)
  
  model = Sequential()
  inputShape = (img_width, img_height, 1)
  model.add(Conv2D(32, (3, 3), activation=&quot;relu&quot;, input_shape=inputShape))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(64, (3, 3), activation=&quot;relu&quot;))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(128, (3, 3), activation=&quot;relu&quot;))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    
  model.add(Flatten())
  model.add(Dense(512, activation=&quot;relu&quot;))
  # to prevent overfitting
  model.add(Dropout(0.5))
  model.add(Dense(3, activation=&quot;softmax&quot;))
  
  opt = Adam(learning_rate=INIT_LR, decay=INIT_LR / EPOCHS)
  
  model.compile(loss=&quot;categorical_crossentropy&quot;, optimizer=opt,
              metrics=[&quot;accuracy&quot;])
  </code></pre>
  <p>This is the <a href="https://i.sstatic.net/PeWVa.png" rel="nofollow noreferrer">Tensorboard graph</a> of the training. The straight line is from another training run.</p>
  <p><strong>Testing/Inference script:</strong></p>
  <pre><code>interpreter = tf.lite.Interpreter(model_path=model_path)
  
  # Load TFLite model and allocate tensors.
  interpreter.allocate_tensors()
  
  # Get input and output tensors.
  input_details = interpreter.get_input_details()
  output_details = interpreter.get_output_details()
  
  height = input_details[0]['shape'][1]
  width = input_details[0]['shape'][2]
  
  labels = [&quot;positive&quot;, &quot;negative&quot;, &quot;initial&quot;]
  
  # load image into numpy array
  image = cv2.imread(image_path)
  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  input_arr = img_to_array(image)
  
  input_arr = np.array([input_arr])
  # normalize values
  input_arr = input_arr / 255.0
  
  interpreter.set_tensor(input_details[0]['index'], input_arr)
  
  interpreter.invoke()
  
  # The function `get_tensor()` returns a copy of the tensor data.
  output_data = interpreter.get_tensor(output_details[0]['index'])
  
  results = np.squeeze(output_data)
  top_k = results.argsort()[-5:][::-1]
  
  print(labels[top_k[0]])
  </code></pre>
  <p>Ideas about where the problem may be are very appreciated. I am stuck with this problem for two months now.</p>
  <p>Thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T00:49:34.780Z
Comments.Text: there is a reason why we use train/validation/test split, and this is one of the reasons... if you use a validation set for hyperparam tuning, you should consider that a "unbiased (possibly overshooting) estimation of your network performance"
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T10:35:42.380Z
Comments.Text: I am using a train/validation split. I edited the post with the LOC. I did not really get how that influences the performance of the network. Shouldn't the network reach a performance around the average validation accuracy?
Comments.UserDisplayName: Sohrab Tawana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T10:44:10.013Z
Comments.Text: Your network seems to include modules, such as `Dropout`, that operate differently during training and inference/testing. For example, dropout is usually active during training, and may drop some connection, whereas all connections are preserved in inference mode. This may be the reason.
Comments.UserDisplayName: mikkola
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T11:46:31.403Z
Comments.Text: I found the issue with a tip from stackoverflow! It was the normalization in the inference script. It is supposed to be input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr). Thanks a lot!
Comments.UserDisplayName: Sohrab Tawana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-18T13:32:03.213Z
Comments.Text: @SohrabTawana if you've found the answer, please write it as an answer and accept it, so that future users can make use of it.
Comments.UserDisplayName: Martino
Tags: <tensorflow><python><image-recognition><keras>
Title: Why does my neural network perform different on the same images during training and testing?
UserDisplayName: Sohrab Tawana
ThreadText: # Why does my neural network perform different on the same images during training and testing?
  
  (Asked by: username_0 on 2022-08-16)
  
  I use tensorflow keras to build a neural network that classifies images of covid-19 rapid tests into three classes (Negative, Positive, Empty).
  During training the tensorboard logs denote a validation accuracy of around 90%. But when I test the network after being trained with the same images it was trained on, the classification performance is way worse (~60%). I observed the same behavior when I trained the network with different images (see section What I have tried).
  During training the images are preprocessed to grayscale and resized before being fed into the model. The batch size is 16.
  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  
  To augment the sparse data that I have (~450 images) I am using the keras.preprocessing.image.ImageDataGenerator and its parameters are: width_shift_range=0.1, brightness_range=[0.9, 1.1], height_shift_range=0.1, zoom_range=0.2, horizontal_flip=True, rotation_range=10, shear_range=0.2, fill_mode="nearest", samplewise_center=True, samplewise_std_normalization=True
  Edit: The images are split into 90/10 training/validation sets.
  (trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.1, random_state=24)
  I am converting the model to tflite because we need it for mobile platforms. I am using this code snippet:
  model = tf.keras.models.load_model(model_path)
  
  converter = tf.lite.TFLiteConverter.from_keras_model(model) # path to the SavedModel directory
  # converter.optimizations = [tf.lite.Optimize.DEFAULT] # optimizations
  tflite_model = converter.convert()
  
  # Save the model.
  with open('rapid_test_strip_cleaned_model.tflite', 'wb') as f:
    f.write(tflite_model)
  
  What I have tried:
  
  crop the images to the strip of the casette, train and test the network again
  check in the testing (inference) script if the labels are correct
  check if the images are converted to grayscale and resized correctly before being fed into the network during testing
  test the model before converting it to tflite, using tensorflow.keras.models
  
  Model:
  img_width, img_height = (256, 256)
  
  model = Sequential()
  inputShape = (img_width, img_height, 1)
  model.add(Conv2D(32, (3, 3), activation="relu", input_shape=inputShape))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(64, (3, 3), activation="relu"))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(128, (3, 3), activation="relu"))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    
  model.add(Flatten())
  model.add(Dense(512, activation="relu"))
  # to prevent overfitting
  model.add(Dropout(0.5))
  model.add(Dense(3, activation="softmax"))
  
  opt = Adam(learning_rate=INIT_LR, decay=INIT_LR / EPOCHS)
  
  model.compile(loss="categorical_crossentropy", optimizer=opt,
              metrics=["accuracy"])
  
  This is the Tensorboard graph of the training. The straight line is from another training run.
  Testing/Inference script:
  interpreter = tf.lite.Interpreter(model_path=model_path)
  
  # Load TFLite model and allocate tensors.
  interpreter.allocate_tensors()
  
  # Get input and output tensors.
  input_details = interpreter.get_input_details()
  output_details = interpreter.get_output_details()
  
  height = input_details[0]['shape'][1]
  width = input_details[0]['shape'][2]
  
  labels = ["positive", "negative", "initial"]
  
  # load image into numpy array
  image = cv2.imread(image_path)
  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  input_arr = img_to_array(image)
  
  input_arr = np.array([input_arr])
  # normalize values
  input_arr = input_arr / 255.0
  
  interpreter.set_tensor(input_details[0]['index'], input_arr)
  
  interpreter.invoke()
  
  # The function `get_tensor()` returns a copy of the tensor data.
  output_data = interpreter.get_tensor(output_details[0]['index'])
  
  results = np.squeeze(output_data)
  top_k = results.argsort()[-5:][::-1]
  
  print(labels[top_k[0]])
  
  Ideas about where the problem may be are very appreciated. I am stuck with this problem for two months now.
  Thank you!
  
  **Tags:** tensorflow, python, image-recognition, keras
  > **Comments:**
  > **username_1**: there is a reason why we use train/validation/test split, and this is one of the reasons... if you use a validation set for hyperparam tuning, you should consider that a "unbiased (possibly overshooting) estimation of your network performance"  
  > **username_0**: I am using a train/validation split. I edited the post with the LOC. I did not really get how that influences the performance of the network. Shouldn't the network reach a performance around the average validation accuracy?  
  > **username_2**: Your network seems to include modules, such as `Dropout`, that operate differently during training and inference/testing. For example, dropout is usually active during training, and may drop some connection, whereas all connections are preserved in inference mode. This may be the reason.  
  > **username_0**: I found the issue with a tip from stackoverflow! It was the normalization in the inference script. It is supposed to be input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr). Thanks a lot!  
  > **username_3**: @SohrabTawana if you've found the answer, please write it as an answer and accept it, so that future users can make use of it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-08-22. Score: 0 | Accepted answer
  
  I found the issue with a tip from stackoverflow!
  
  The problem is the normalization in the inference script. It is supposed to be `input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr)`.
  
  
  ---
  
  