First of all, you probably need another transform for the validation set, by removing
transforms.Resize(320),
transforms.CenterCrop(299)

I am not sure if this is part of the problem. Moreover, you need to set Shuffle to False in the val loader.