edshkim98:

My validation data also has similar distribution as the training dataset.


I think I misunderstood the use case and thought you might have a slight difference between the training and validation datasets, but it seems the difference is between training+validation vs. test?
I don’t think adding data augmentation to the validation set is a good idea, but also the difference of the test dataset is concerning. A common way would be to decrease the test set and use some of it in the training and validation set. However, if your test set is small or if you don’t have the targets, this won’t be easily possible.
In that case unfortunately I don’t know what the best approach would be to generalize to a “new” data domain.