You have a low sample size. I would really up the augmentation and set the dropout to 50%. Use as many transformers as you can before putting the images into the model.
I.e. ColorJitter, RandomAffine, RandomResizedCrop, RandomHorizontalFlip, RandomPerspective, RandomRotation, GaussianBlur, RandomInvert, RandomPosterize, RandomSolarize, etc.
https://pytorch.org/vision/stable/transforms.html
You can’t go wrong with too many transforms, especially with low data. Nvidia’s research team has found aggressive transforms produce excellent results for ML vision.
  

      The Official NVIDIA Blog – 7 Dec 20
  

  
    

NVIDIA Research Achieves AI Training Breakthrough | NVIDIA Blog 2

  Data augmentation technique enables GAN to emulate artwork from a limited dataset — and opens up potential applications in healthcare.

  
    Est. reading time: 4 minutes
  

  

  
    
    
  

  


One other question, how many parameters are you using for the entire model? For example:
print(sum(p.numel() for p in model.parameters()))