Right, I switched from using a pretrained (on Imagenet) Resnet50 to a Resnet18, and that lowered the overfitting, so that my trainset Top1 accuracy is now around 58% (down from 69%).
Would increasing data augmentation rate (from 16x to say 96x) decrease overfitting further?
One other thing about the nature of my dataset-there is a severe class size imbalance as well across the 110 classes-could this be contributing to the overfitting as well? What would be a good solution to this then-perhaps clustering the smaller classes into aggregate classes to match the size of the larger classes? Then perhaps a second net could be used where the aggregate classes are then broken down back into their original classes for more fine-grained classification.
Lastly, what would you say is a reasonable amount of overfitting in terms of the trainset-testset difference in accuracy measurement? 5%?
Thanks