I have been training resnet 18 as well as resnet50 model on my dataset(which contains about 1500 samples).
When it comes to predicting output labels of dimension 73x4 and 1x199,the models tend to do well,but for output vectors of dimension 1x6 or for classification purposes,I have noticed that there is huge overfitting.
With training loss being 0.03 and validation loss being 0.24
What steps can I take to avoid overfitting.
Should I apply data augmentation on my images,or should I go for a different pretrained model like efficientnet/vgg.
Note I tried using vgg16 ,but no effect in performance was seen.
Enclosed is the loss curve

plot712×568 13.2 KB