title: Accuracy low on validation set using vgg16
body: <p>I'm doing dog breed classification for a project and I encounter a major issue I have no idea how to solve it.</p>
  
  <p>The dataset is the images of dogs provided by <a href="http://vision.stanford.edu/aditya86/ImageNetDogs/" rel="nofollow noreferrer">Stanford Dogs dataset</a>.</p>
  
  <p>I do a data augmentation with keras :</p>
  
  <pre><code>from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
  from keras import optimizers
  from keras.callbacks import History 
  from keras.applications import vgg16
  
  batch_size = 16
  
  # this is the augmentation configuration I will use for training
   train_datagen = ImageDataGenerator(rotation_range=20,
                                 zoom_range=0.2,
                                 horizontal_flip=True,
                                 fill_mode='nearest',
                                 preprocessing_function=vgg16.preprocess_input)
  
   # This is the augmentation configuration I will use for testing/validation... just a rescale
   test_datagen = ImageDataGenerator(rescale=1./255)
  
   # This is the generator which will read pictures found in my training subset
   train_generator = train_datagen.flow_from_directory('../data/train/',
                                                  target_size = (224, 224),
                                                  batch_size = batch_size,
                                                  shuffle=True,
                                                  class_mode = 'categorical',
                                                  seed=42)
  
    # This is the generator for validation data
    validation_generator = test_datagen.flow_from_directory('../data/validation/',
                                                      target_size = (224, 224),
                                                      batch_size = batch_size,
                                                      class_mode = 'categorical')
  </code></pre>
  
  <p>Then I'm using transfer learning with VGG16 to train my model :</p>
  
  <pre><code>from keras.applications.vgg16 import VGG16
  from keras.layers import Input, Dense, Flatten, GlobalAveragePooling2D
  from keras.models import Model, Sequential
  
  pretrained_model = VGG16(weights="imagenet", include_top=False, input_shape=(224, 224, 3))
  
  for layer in pretrained_model.layers:
      layer.trainable = False
  
  x = pretrained_model.output
  x = GlobalAveragePooling2D()(x)
  x = Dense(1024, activation="relu")(x)
  predictions = Dense(120, activation='softmax')(x)
  
  model = Model(inputs = pretrained_model.input, outputs=predictions) 
  
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  </code></pre>
  
  <p>When I trained the model it seems like it's overfitting :
  I got 0.99 accuracy on train and 0.23 on validation.</p>
  
  <p>When looking to the images in the train and validation set there are no apparent reasons it gives me such a bad accuracy for the validation.</p>
  
  <p>What I've done is checked manually what breed the model predicts for several images (more than 50) in the vaidation and test and more than 80% are corrects predictions. </p>
  
  <p>So I'm thinking there is a problem on the validation accuracy but have no idea how to fix it.</p>
  
  <p>Thanks for your help !</p>