Your data augmentation might be too aggressive and might not represent the underlying data domain properly, so that the model learns the training set but gets worse in classifying the validation samples.