title: Keras validation loss is calculating incorrectly
body: <p>My model trains fine on a CPU machine but I am running into an issue when trying to rerun it on our cluster (using a single GPU and the same dataset). When training on a GPU machine validation loss and accuracy are not improving from epoch to epoch (see below).This was not the case on a CPU machine (I was able to achieve validation accuracy ~0.8 after 20 epochs)</p>
  
  <p><strong>Details:</strong></p>
  
  <p>Keras 2.1.3</p>
  
  <p>TensforFlow backend</p>
  
  <p>70/20/10 train/dev/test</p>
  
  <p>~ 7000 images</p>
  
  <p>model is based on ResNet50</p>
  
  <p><strong>Code</strong></p>
  
  <pre><code>    import sys
      import math
      import os
      import glob
  
      from keras import applications
      from keras.preprocessing.image import ImageDataGenerator
      from keras import optimizers
      from keras.models import Sequential, Model 
      from keras.layers import Flatten, Dense
      from keras import backend as k 
      from keras.callbacks import ModelCheckpoint, CSVLogger, EarlyStopping
  
      ############ Training parameters ##################
      img_width, img_height = 224, 224
      batch_size = 32
      epochs = 100
  
  
      ############ Define the data ##################
      train_data_dir = '/mnt/data/train'
      validation_data_dir = '/mnt/data/validate'
  
      train_data_dir_class1 = os.path.join(train_data_dir,'class1', '*.jpg')
      train_data_dir_class2 = os.path.join(train_data_dir, 'class2', '*.jpg')
  
      validation_data_dir_class1 = os.path.join(validation_data_dir, 'class1', '*.jpg')
      validation_data_dir_class2 = os.path.join(validation_data_dir, 'class2', '*.jpg')
  
      # number of training and validation samples
      nb_train_samples = len(glob.glob(train_data_dir_class1)) + len(glob.glob(train_data_dir_class2))
      nb_validation_samples = len(glob.glob(validation_data_dir_class1)) + len(glob.glob(validation_data_dir_class2))
  
  
      ############ Define the model ##################
      model = applications.resnet50.ResNet50(weights = "imagenet",
                                             include_top = False,
                                             input_shape = (img_width, img_height, 3))
  
      for layer in model.layers:
          layer.trainable = False
  
      # Adding a FC layer
      x = model.output
      x = Flatten()(x)
      predictions = Dense(1, activation = "sigmoid")(x)
  
      # creating the final model 
      model_final = Model(inputs = model.input, outputs = predictions)
  
      # compile the model 
      model_final.compile(loss = "binary_crossentropy",
                          optimizer = optimizers.Adam(lr = 0.001,
                                                      beta_1 = 0.9,
                                                      beta_2 = 0.999,
                                                      epsilon = 1e-10),
                          metrics = ["accuracy"])
  
      # train and test generators 
      train_datagen = ImageDataGenerator(rescale = 1./255,
                                         horizontal_flip = True,
                                         fill_mode = "nearest",
                                         zoom_range = 0.3,
                                         width_shift_range = 0.3,
                                         height_shift_range = 0.3,
                                         rotation_range = 30)
  
      test_datagen = ImageDataGenerator(rescale = 1./255)
  
      train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                          target_size = (img_height, img_width),
                                                          batch_size = batch_size,
                                                          class_mode = "binary",
                                                          seed = 2018)
  
      validation_generator = test_datagen.flow_from_directory(validation_data_dir,
                                                              target_size = (img_height, img_width),
                                                              class_mode = "binary",
                                                              seed = 2018)
  
      early = EarlyStopping(monitor = 'val_loss', min_delta = 10e-5, patience = 10, verbose = 1, mode = 'auto')
      performance_log = CSVLogger('/mnt/results/vanilla_model_log.csv', separator = ',', append = False)
  
      # Train the model
      model_final.fit_generator(generator = train_generator,
                                steps_per_epoch = math.ceil(train_generator.samples / batch_size),
                                epochs = epochs,
                                validation_data = validation_generator,
                                validation_steps = math.ceil(validation_generator.samples / batch_size),
                                callbacks = [early, performance_log])
  
      # Save the model
      model_final.save('/mnt/results/vanilla_model.h5')
  </code></pre>
  
  <p><strong>Training Log</strong></p>
  
  <pre><code>Epoch 1/100
  151/151 [==============================] - 237s 2s/step - loss: 0.7234 - acc: 0.5240 - val_loss: 0.9899 - val_acc: 0.5425
  Epoch 2/100
  151/151 [==============================] - 65s 428ms/step - loss: 0.6491 - acc: 0.6228 - val_loss: 1.0248 - val_acc: 0.5425
  Epoch 3/100
  151/151 [==============================] - 65s 429ms/step - loss: 0.6091 - acc: 0.6648 - val_loss: 1.0377 - val_acc: 0.5425
  Epoch 4/100
  151/151 [==============================] - 64s 426ms/step - loss: 0.5829 - acc: 0.6968 - val_loss: 1.0459 - val_acc: 0.5425
  Epoch 5/100
  151/151 [==============================] - 64s 427ms/step - loss: 0.5722 - acc: 0.7070 - val_loss: 1.0472 - val_acc: 0.5425
  Epoch 6/100
  151/151 [==============================] - 64s 427ms/step - loss: 0.5582 - acc: 0.7166 - val_loss: 1.0501 - val_acc: 0.5425
  Epoch 7/100
  151/151 [==============================] - 64s 424ms/step - loss: 0.5535 - acc: 0.7188 - val_loss: 1.0492 - val_acc: 0.5425
  Epoch 8/100
  151/151 [==============================] - 64s 426ms/step - loss: 0.5377 - acc: 0.7287 - val_loss: 1.0209 - val_acc: 0.5425
  Epoch 9/100
  151/151 [==============================] - 64s 425ms/step - loss: 0.5328 - acc: 0.7368 - val_loss: 1.0062 - val_acc: 0.5425
  Epoch 10/100
  151/151 [==============================] - 65s 432ms/step - loss: 0.5296 - acc: 0.7381 - val_loss: 1.0016 - val_acc: 0.5425
  Epoch 11/100
  151/151 [==============================] - 65s 430ms/step - loss: 0.5231 - acc: 0.7419 - val_loss: 1.0021 - val_acc: 0.5425
  </code></pre>
  
  <p>Since I was able to get good results on a CPU machine, I hypothesized that validation loss/accuracy must be calculated incorrectly at the end of each epoch. To test this theory I used train set as validation set: if validation loss/accuracy is calculated correctly we should see roughly the same values for train and validation loss and accuracy. As you may see below, validation loss values are not the same as training loss values, which makes me believe validation loss is calculated incorrectly at the end of each epoch. <strong>Why does it happen? What are the possible solutions?</strong></p>
  
  <p><strong>Modified Code</strong></p>
  
  <pre><code>    import sys
      import math
      import os
      import glob
  
      from keras import applications
      from keras.preprocessing.image import ImageDataGenerator
      from keras import optimizers
      from keras.models import Sequential, Model 
      from keras.layers import Flatten, Dense
      from keras import backend as k 
      from keras.callbacks import ModelCheckpoint, CSVLogger, EarlyStopping
  
      ############ Training parameters ##################
      img_width, img_height = 224, 224
      batch_size = 32
      epochs = 100
  
  
      ############ Define the data ##################
      train_data_dir = '/mnt/data/train'
      validation_data_dir = '/mnt/data/train' # redefined validation set to test accuracy of validation loss/accuracy calculations
  
      train_data_dir_class1 = os.path.join(train_data_dir,'class1', '*.jpg')
      train_data_dir_class2 = os.path.join(train_data_dir, 'class2', '*.jpg')
  
      validation_data_dir_class1 = os.path.join(validation_data_dir, 'class1', '*.jpg')
      validation_data_dir_class2 = os.path.join(validation_data_dir, 'class2', '*.jpg')
  
      # number of training and validation samples
      nb_train_samples = len(glob.glob(train_data_dir_class1)) + len(glob.glob(train_data_dir_class2))
      nb_validation_samples = len(glob.glob(validation_data_dir_class1)) + len(glob.glob(validation_data_dir_class2))
  
  
      ############ Define the model ##################
      model = applications.resnet50.ResNet50(weights = "imagenet",
                                             include_top = False,
                                             input_shape = (img_width, img_height, 3))
  
      for layer in model.layers:
          layer.trainable = False
  
      # Adding a FC layer
      x = model.output
      x = Flatten()(x)
      predictions = Dense(1, activation = "sigmoid")(x)
  
      # creating the final model 
      model_final = Model(inputs = model.input, outputs = predictions)
  
      # compile the model 
      model_final.compile(loss = "binary_crossentropy",
                          optimizer = optimizers.Adam(lr = 0.001,
                                                      beta_1 = 0.9,
                                                      beta_2 = 0.999,
                                                      epsilon = 1e-10),
                          metrics = ["accuracy"])
  
      # train and test generators 
      train_datagen = ImageDataGenerator(rescale = 1./255,
                                         horizontal_flip = True,
                                         fill_mode = "nearest",
                                         zoom_range = 0.3,
                                         width_shift_range = 0.3,
                                         height_shift_range = 0.3,
                                         rotation_range = 30)
  
      test_datagen = ImageDataGenerator(rescale = 1./255)
  
      train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                          target_size = (img_height, img_width),
                                                          batch_size = batch_size,
                                                          class_mode = "binary",
                                                          seed = 2018)
  
      validation_generator = test_datagen.flow_from_directory(validation_data_dir,
                                                              target_size = (img_height, img_width),
                                                              class_mode = "binary",
                                                              seed = 2018)
  
      early = EarlyStopping(monitor = 'val_loss', min_delta = 10e-5, patience = 10, verbose = 1, mode = 'auto')
      performance_log = CSVLogger('/mnt/results/vanilla_model_log.csv', separator = ',', append = False)
  
      # Train the model
      model_final.fit_generator(generator = train_generator,
                                steps_per_epoch = math.ceil(train_generator.samples / batch_size),
                                epochs = epochs,
                                validation_data = validation_generator,
                                validation_steps = math.ceil(validation_generator.samples / batch_size),
                                callbacks = [early, performance_log])
  
      # Save the model
      model_final.save('/mnt/results/vanilla_model.h5')
  </code></pre>
  
  <p><strong>Training log for the modified code:</strong></p>
  
  <pre><code>Epoch 1/100
  151/151 [==============================] - 251s 2s/step - loss: 0.6804 - acc: 0.5910 - val_loss: 0.6923 - val_acc: 0.5469
  Epoch 2/100
  151/151 [==============================] - 87s 578ms/step - loss: 0.6258 - acc: 0.6523 - val_loss: 0.6938 - val_acc: 0.5469
  Epoch 3/100
  151/151 [==============================] - 88s 580ms/step - loss: 0.5946 - acc: 0.6874 - val_loss: 0.7001 - val_acc: 0.5469
  Epoch 4/100
  151/151 [==============================] - 88s 580ms/step - loss: 0.5718 - acc: 0.7086 - val_loss: 0.7036 - val_acc: 0.5469
  Epoch 5/100
  151/151 [==============================] - 87s 578ms/step - loss: 0.5634 - acc: 0.7157 - val_loss: 0.7067 - val_acc: 0.5469
  Epoch 6/100
  151/151 [==============================] - 87s 578ms/step - loss: 0.5467 - acc: 0.7243 - val_loss: 0.7099 - val_acc: 0.5469
  Epoch 7/100
  151/151 [==============================] - 87s 578ms/step - loss: 0.5392 - acc: 0.7317 - val_loss: 0.7096 - val_acc: 0.5469
  Epoch 8/100
  151/151 [==============================] - 87s 578ms/step - loss: 0.5287 - acc: 0.7387 - val_loss: 0.7083 - val_acc: 0.5469
  Epoch 9/100
  151/151 [==============================] - 87s 575ms/step - loss: 0.5306 - acc: 0.7385 - val_loss: 0.7088 - val_acc: 0.5469
  Epoch 10/100
  151/151 [==============================] - 87s 577ms/step - loss: 0.5303 - acc: 0.7318 - val_loss: 0.7111 - val_acc: 0.5469
  Epoch 11/100
  151/151 [==============================] - 87s 578ms/step - loss: 0.5157 - acc: 0.7474 - val_loss: 0.7143 - val_acc: 0.5469
  </code></pre>