Big Models can overfit and so it if not a gurantee a bigger model will always perform better.
I see you have tranforms (resize and centercrop) against valid and test set. I do not think you intend to do TTA. So, can you drop these two transforms and see how your model (ResNet18 for start) performs?