Answers.Body: <p>I found the issue with a tip from stackoverflow!</p>
  <p>The problem is the normalization in the inference script. It is supposed to be <code>input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr)</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-22T09:23:04.273Z
Answers.LastActivityDate: 2022-08-22T09:23:04.273Z
Answers.UserDisplayName: Sohrab Tawana
Body: <p>I use <code>tensorflow keras</code> to build a neural network that classifies images of covid-19 rapid tests into three classes (<a href="https://i.sstatic.net/M53bS.jpg" rel="nofollow noreferrer">Negative</a>, <a href="https://i.sstatic.net/uqiRV.jpg" rel="nofollow noreferrer">Positive</a>, <a href="https://i.sstatic.net/SBLrd.jpg" rel="nofollow noreferrer">Empty</a>).</p>
  <p>During training the tensorboard logs denote a validation accuracy of around 90%. But when I test the network after being trained with the same images it was trained on, the classification performance is way worse (~60%). I observed the same behavior when I trained the network with different images (see section <code>What I have tried</code>).</p>
  <p>During training the images are preprocessed to grayscale and resized before being fed into the model. The batch size is 16.</p>
  <pre><code>image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  </code></pre>
  <p>To augment the sparse data that I have (~450 images) I am using the <code>keras.preprocessing.image.ImageDataGenerator</code> and its parameters are: <code>width_shift_range=0.1</code>, <code>brightness_range=[0.9, 1.1]</code>, <code>height_shift_range=0.1</code>, <code>zoom_range=0.2</code>, <code>horizontal_flip=True</code>, <code>rotation_range=10</code>, <code>shear_range=0.2</code>, <code>fill_mode=&quot;nearest&quot;</code>, <code>samplewise_center=True</code>, <code>samplewise_std_normalization=True</code></p>
  <p>Edit: The images are split into 90/10 training/validation sets.</p>
  <p><code>(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.1, random_state=24)</code></p>
  <p>I am converting the model to tflite because we need it for mobile platforms. I am using this code snippet:</p>
  <pre><code>model = tf.keras.models.load_model(model_path)
  
  converter = tf.lite.TFLiteConverter.from_keras_model(model) # path to the SavedModel directory
  # converter.optimizations = [tf.lite.Optimize.DEFAULT] # optimizations
  tflite_model = converter.convert()
  
  # Save the model.
  with open('rapid_test_strip_cleaned_model.tflite', 'wb') as f:
    f.write(tflite_model)
  </code></pre>
  <p><strong>What I have tried:</strong></p>
  <ul>
  <li>crop the images to the <a href="https://i.sstatic.net/CCKtS.jpg" rel="nofollow noreferrer">strip</a> of the casette, train and test the network again</li>
  <li>check in the testing (inference) script if the labels are correct</li>
  <li>check if the images are converted to grayscale and resized correctly before being fed into the network during testing</li>
  <li>test the model before converting it to tflite, using <code>tensorflow.keras.models</code></li>
  </ul>
  <p><strong>Model:</strong></p>
  <pre><code>img_width, img_height = (256, 256)
  
  model = Sequential()
  inputShape = (img_width, img_height, 1)
  model.add(Conv2D(32, (3, 3), activation=&quot;relu&quot;, input_shape=inputShape))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(64, (3, 3), activation=&quot;relu&quot;))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(128, (3, 3), activation=&quot;relu&quot;))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    
  model.add(Flatten())
  model.add(Dense(512, activation=&quot;relu&quot;))
  # to prevent overfitting
  model.add(Dropout(0.5))
  model.add(Dense(3, activation=&quot;softmax&quot;))
  
  opt = Adam(learning_rate=INIT_LR, decay=INIT_LR / EPOCHS)
  
  model.compile(loss=&quot;categorical_crossentropy&quot;, optimizer=opt,
              metrics=[&quot;accuracy&quot;])
  </code></pre>
  <p>This is the <a href="https://i.sstatic.net/PeWVa.png" rel="nofollow noreferrer">Tensorboard graph</a> of the training. The straight line is from another training run.</p>
  <p><strong>Testing/Inference script:</strong></p>
  <pre><code>interpreter = tf.lite.Interpreter(model_path=model_path)
  
  # Load TFLite model and allocate tensors.
  interpreter.allocate_tensors()
  
  # Get input and output tensors.
  input_details = interpreter.get_input_details()
  output_details = interpreter.get_output_details()
  
  height = input_details[0]['shape'][1]
  width = input_details[0]['shape'][2]
  
  labels = [&quot;positive&quot;, &quot;negative&quot;, &quot;initial&quot;]
  
  # load image into numpy array
  image = cv2.imread(image_path)
  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  input_arr = img_to_array(image)
  
  input_arr = np.array([input_arr])
  # normalize values
  input_arr = input_arr / 255.0
  
  interpreter.set_tensor(input_details[0]['index'], input_arr)
  
  interpreter.invoke()
  
  # The function `get_tensor()` returns a copy of the tensor data.
  output_data = interpreter.get_tensor(output_details[0]['index'])
  
  results = np.squeeze(output_data)
  top_k = results.argsort()[-5:][::-1]
  
  print(labels[top_k[0]])
  </code></pre>
  <p>Ideas about where the problem may be are very appreciated. I am stuck with this problem for two months now.</p>
  <p>Thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T00:49:34.780Z
Comments.Text: there is a reason why we use train/validation/test split, and this is one of the reasons... if you use a validation set for hyperparam tuning, you should consider that a "unbiased (possibly overshooting) estimation of your network performance"
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T10:35:42.380Z
Comments.Text: I am using a train/validation split. I edited the post with the LOC. I did not really get how that influences the performance of the network. Shouldn't the network reach a performance around the average validation accuracy?
Comments.UserDisplayName: Sohrab Tawana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T10:44:10.013Z
Comments.Text: Your network seems to include modules, such as `Dropout`, that operate differently during training and inference/testing. For example, dropout is usually active during training, and may drop some connection, whereas all connections are preserved in inference mode. This may be the reason.
Comments.UserDisplayName: mikkola
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T11:46:31.403Z
Comments.Text: I found the issue with a tip from stackoverflow! It was the normalization in the inference script. It is supposed to be input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr). Thanks a lot!
Comments.UserDisplayName: Sohrab Tawana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-18T13:32:03.213Z
Comments.Text: @SohrabTawana if you've found the answer, please write it as an answer and accept it, so that future users can make use of it.
Comments.UserDisplayName: Martino
Tags: <tensorflow><python><image-recognition><keras>
Title: Why does my neural network perform different on the same images during training and testing?
UserDisplayName: Sohrab Tawana
ThreadText: # Why does my neural network perform different on the same images during training and testing?
  
  (Asked by: username_0 on 2022-08-16)
  
  I use `tensorflow keras` to build a neural network that classifies images of covid-19 rapid tests into three classes ([Negative](https://i.sstatic.net/M53bS.jpg), [Positive](https://i.sstatic.net/uqiRV.jpg), [Empty](https://i.sstatic.net/SBLrd.jpg)).
  
  During training the tensorboard logs denote a validation accuracy of around 90%. But when I test the network after being trained with the same images it was trained on, the classification performance is way worse (~60%). I observed the same behavior when I trained the network with different images (see section `What I have tried`).
  
  During training the images are preprocessed to grayscale and resized before being fed into the model. The batch size is 16.
  
  ```
  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  
  ```
  
  To augment the sparse data that I have (~450 images) I am using the `keras.preprocessing.image.ImageDataGenerator` and its parameters are: `width_shift_range=0.1`, `brightness_range=[0.9, 1.1]`, `height_shift_range=0.1`, `zoom_range=0.2`, `horizontal_flip=True`, `rotation_range=10`, `shear_range=0.2`, `fill_mode="nearest"`, `samplewise_center=True`, `samplewise_std_normalization=True`
  
  Edit: The images are split into 90/10 training/validation sets.
  
  `(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.1, random_state=24)`
  
  I am converting the model to tflite because we need it for mobile platforms. I am using this code snippet:
  
  ```
  model = tf.keras.models.load_model(model_path)
  
  converter = tf.lite.TFLiteConverter.from_keras_model(model) # path to the SavedModel directory
  # converter.optimizations = [tf.lite.Optimize.DEFAULT] # optimizations
  tflite_model = converter.convert()
  
  # Save the model.
  with open('rapid_test_strip_cleaned_model.tflite', 'wb') as f:
    f.write(tflite_model)
  
  ```
  
  **What I have tried:**
  
  * crop the images to the [strip](https://i.sstatic.net/CCKtS.jpg) of the casette, train and test the network again
  * check in the testing (inference) script if the labels are correct
  * check if the images are converted to grayscale and resized correctly before being fed into the network during testing
  * test the model before converting it to tflite, using `tensorflow.keras.models`
  
  **Model:**
  
  ```
  img_width, img_height = (256, 256)
  
  model = Sequential()
  inputShape = (img_width, img_height, 1)
  model.add(Conv2D(32, (3, 3), activation="relu", input_shape=inputShape))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(64, (3, 3), activation="relu"))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  model.add(Conv2D(128, (3, 3), activation="relu"))
  # to prevent overfitting
  model.add(Dropout(0.25))
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    
  model.add(Flatten())
  model.add(Dense(512, activation="relu"))
  # to prevent overfitting
  model.add(Dropout(0.5))
  model.add(Dense(3, activation="softmax"))
  
  opt = Adam(learning_rate=INIT_LR, decay=INIT_LR / EPOCHS)
  
  model.compile(loss="categorical_crossentropy", optimizer=opt,
              metrics=["accuracy"])
  
  ```
  
  This is the [Tensorboard graph](https://i.sstatic.net/PeWVa.png) of the training. The straight line is from another training run.
  
  **Testing/Inference script:**
  
  ```
  interpreter = tf.lite.Interpreter(model_path=model_path)
  
  # Load TFLite model and allocate tensors.
  interpreter.allocate_tensors()
  
  # Get input and output tensors.
  input_details = interpreter.get_input_details()
  output_details = interpreter.get_output_details()
  
  height = input_details[0]['shape'][1]
  width = input_details[0]['shape'][2]
  
  labels = ["positive", "negative", "initial"]
  
  # load image into numpy array
  image = cv2.imread(image_path)
  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  image = cv2.resize(image, (height, width))
  input_arr = img_to_array(image)
  
  input_arr = np.array([input_arr])
  # normalize values
  input_arr = input_arr / 255.0
  
  interpreter.set_tensor(input_details[0]['index'], input_arr)
  
  interpreter.invoke()
  
  # The function `get_tensor()` returns a copy of the tensor data.
  output_data = interpreter.get_tensor(output_details[0]['index'])
  
  results = np.squeeze(output_data)
  top_k = results.argsort()[-5:][::-1]
  
  print(labels[top_k[0]])
  
  ```
  
  Ideas about where the problem may be are very appreciated. I am stuck with this problem for two months now.
  
  Thank you!
  
  **Tags:** tensorflow, python, image-recognition, keras
  
  > **Comments:**
  > **username_1**: there is a reason why we use train/validation/test split, and this is one of the reasons... if you use a validation set for hyperparam tuning, you should consider that a "unbiased (possibly overshooting) estimation of your network performance"  
  > **username_0**: I am using a train/validation split. I edited the post with the LOC. I did not really get how that influences the performance of the network. Shouldn't the network reach a performance around the average validation accuracy?  
  > **username_2**: Your network seems to include modules, such as `Dropout`, that operate differently during training and inference/testing. For example, dropout is usually active during training, and may drop some connection, whereas all connections are preserved in inference mode. This may be the reason.  
  > **username_0**: I found the issue with a tip from stackoverflow! It was the normalization in the inference script. It is supposed to be input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr). Thanks a lot!  
  > **username_3**: @username_0 if you've found the answer, please write it as an answer and accept it, so that future users can make use of it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-08-22. Score: 0 | Accepted answer
  
  I found the issue with a tip from stackoverflow!
  
  The problem is the normalization in the inference script. It is supposed to be `input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr)`.
  
  
  ---
  
  