Hi,
I’ve build a new model using VGG16 architecture, and trained only the fully connected layer in order to classify gender of a person. During training, the accuracy over the training set of 10k photos is reaching 94% and over the validation set around 82%  .
The problem appear after the training is over, i’m doing a simple test by arbitrary choosing some photos from the training dataset,  and the results just seems wrong, the accuracy i’m getting is somewhere around 50%. If i got 94% accuracy on the model, and using the same dataset the model was trained on, that means it should be wrong for only 6 out of 100 photos, but that is not the case.
I didn’t provide any code because I didn’t know what could be relavent, so if anything is needed i’ll add it.
I’ll appreciate any thoughts on why and how this could happen.