You really need to check that you have uniform sampled between the train and eval to cover the same augmentation hyperparams range.
I think also that a 500 sample dataset it could be a little bit small also with augmentation.