Hi Karthik, Thank you so much for the reply. As you said I have removed the augmentations from the phases and used Resnet 18 as the model. I used them as the labels seemed imbalanced. Now the validation accuracy increased to 83% from 71% . However the training accuracy is always at an average of 99%( it even showed 100% accuracy in a few epochs).The model is definitely overfitting at this point.  How can I have a realistic training accuracy and increase the validation accuracy as well? Thanks.
This is my epoch history btw.
Epoch 0/69
----------
train Loss: 0.4579 Acc: 0.8418
val Loss: 0.7711 Acc: 0.7895

Epoch 1/69
----------
train Loss: 0.3891 Acc: 0.8604
val Loss: 0.7123 Acc: 0.8086

Epoch 2/69
----------
train Loss: 0.3274 Acc: 0.8912
val Loss: 1.1504 Acc: 0.7492

Epoch 3/69
----------
train Loss: 0.2924 Acc: 0.8968
val Loss: 0.9519 Acc: 0.8013

Epoch 4/69
----------
train Loss: 0.2063 Acc: 0.9292
val Loss: 1.3863 Acc: 0.7260

Epoch 5/69
----------
train Loss: 0.1951 Acc: 0.9304
val Loss: 1.0404 Acc: 0.7761

Epoch 6/69
----------
train Loss: 0.1559 Acc: 0.9453
val Loss: 0.8151 Acc: 0.8204

Epoch 7/69
----------
train Loss: 0.1608 Acc: 0.9455
val Loss: 1.9316 Acc: 0.7549

Epoch 8/69
----------
train Loss: 0.1283 Acc: 0.9557
val Loss: 1.5562 Acc: 0.7585

Epoch 9/69
----------
train Loss: 0.1162 Acc: 0.9616
val Loss: 1.6801 Acc: 0.7023

Epoch 10/69
----------
train Loss: 0.0999 Acc: 0.9665
val Loss: 1.0993 Acc: 0.8065

Epoch 11/69
----------
train Loss: 0.0774 Acc: 0.9742
val Loss: 1.1724 Acc: 0.7967

Epoch 12/69
----------
train Loss: 0.0864 Acc: 0.9698
val Loss: 1.2067 Acc: 0.8122

Epoch 13/69
----------
train Loss: 0.0795 Acc: 0.9738
val Loss: 1.0400 Acc: 0.8091

Epoch 14/69
----------
train Loss: 0.0600 Acc: 0.9796
val Loss: 1.8943 Acc: 0.7085

Epoch 15/69
----------
train Loss: 0.0581 Acc: 0.9804
val Loss: 1.3067 Acc: 0.7843

Epoch 16/69
----------
train Loss: 0.0624 Acc: 0.9802
val Loss: 1.3855 Acc: 0.8080

Epoch 17/69
----------
train Loss: 0.0624 Acc: 0.9796
val Loss: 1.0680 Acc: 0.8194

Epoch 18/69
----------
train Loss: 0.0443 Acc: 0.9864
val Loss: 1.2256 Acc: 0.8137

Epoch 19/69
----------
train Loss: 0.0242 Acc: 0.9908
val Loss: 1.5738 Acc: 0.8091

Epoch 20/69
----------
train Loss: 0.0314 Acc: 0.9915
val Loss: 1.3549 Acc: 0.8075

Epoch 21/69
----------
train Loss: 0.0345 Acc: 0.9888
val Loss: 1.3210 Acc: 0.8019

Epoch 22/69
----------
train Loss: 0.0364 Acc: 0.9881
val Loss: 1.7013 Acc: 0.7967

Epoch 23/69
----------
train Loss: 0.0312 Acc: 0.9899
val Loss: 1.5513 Acc: 0.8091

Epoch 24/69
----------
train Loss: 0.0579 Acc: 0.9804
val Loss: 1.4634 Acc: 0.8111

Epoch 25/69
----------
train Loss: 0.0345 Acc: 0.9899
val Loss: 1.1516 Acc: 0.8070

Epoch 26/69
----------
train Loss: 0.0225 Acc: 0.9924
val Loss: 1.6052 Acc: 0.7962

Epoch 27/69
----------
train Loss: 0.0102 Acc: 0.9967
val Loss: 1.3944 Acc: 0.8189

Epoch 28/69
----------
train Loss: 0.0064 Acc: 0.9979
val Loss: 1.7504 Acc: 0.8127

Epoch 29/69
----------
train Loss: 0.0315 Acc: 0.9910
val Loss: 1.4341 Acc: 0.8204

Epoch 30/69
----------
train Loss: 0.0270 Acc: 0.9910
val Loss: 1.5566 Acc: 0.8013

Epoch 31/69
----------
train Loss: 0.0095 Acc: 0.9972
val Loss: 1.2792 Acc: 0.8344

Epoch 32/69
----------
train Loss: 0.0094 Acc: 0.9968
val Loss: 1.7946 Acc: 0.7879

Epoch 33/69
----------
train Loss: 0.0255 Acc: 0.9911
val Loss: 1.4167 Acc: 0.8148

Epoch 34/69
----------
train Loss: 0.0244 Acc: 0.9935
val Loss: 1.5633 Acc: 0.7585

Epoch 35/69
----------
train Loss: 0.0110 Acc: 0.9972
val Loss: 1.4525 Acc: 0.8179

Epoch 36/69
----------
train Loss: 0.0036 Acc: 0.9987
val Loss: 1.3777 Acc: 0.8225

Epoch 37/69
----------
train Loss: 0.0014 Acc: 0.9997
val Loss: 1.3799 Acc: 0.8204

Epoch 38/69
----------
train Loss: 0.0039 Acc: 0.9986
val Loss: 1.4773 Acc: 0.8065

Epoch 39/69
----------
train Loss: 0.0060 Acc: 0.9979
val Loss: 1.7012 Acc: 0.8060

Epoch 40/69
----------
train Loss: 0.0060 Acc: 0.9981
val Loss: 1.5426 Acc: 0.8127

Epoch 41/69
----------
train Loss: 0.0042 Acc: 0.9989
val Loss: 1.4538 Acc: 0.8189

Epoch 42/69
----------
train Loss: 0.0021 Acc: 0.9994
val Loss: 1.5516 Acc: 0.8220

Epoch 43/69
----------
train Loss: 0.0067 Acc: 0.9975
val Loss: 1.7034 Acc: 0.8039

Epoch 44/69
----------
train Loss: 0.0182 Acc: 0.9946
val Loss: 1.4298 Acc: 0.8277

Epoch 45/69
----------
train Loss: 0.0342 Acc: 0.9889
val Loss: 1.3819 Acc: 0.8132

Epoch 46/69
----------
train Loss: 0.0155 Acc: 0.9953
val Loss: 1.4299 Acc: 0.8142

Epoch 47/69
----------
train Loss: 0.0094 Acc: 0.9968
val Loss: 1.5808 Acc: 0.7993

Epoch 48/69
----------
train Loss: 0.0087 Acc: 0.9973
val Loss: 1.2893 Acc: 0.8323

Epoch 49/69
----------
train Loss: 0.0068 Acc: 0.9978
val Loss: 1.6876 Acc: 0.7951

Epoch 50/69
----------
train Loss: 0.0076 Acc: 0.9981
val Loss: 1.2886 Acc: 0.8235

Epoch 51/69
----------
train Loss: 0.0183 Acc: 0.9938
val Loss: 1.4195 Acc: 0.8199

Epoch 52/69
----------
train Loss: 0.0167 Acc: 0.9951
val Loss: 1.2933 Acc: 0.8209

Epoch 53/69
----------
train Loss: 0.0055 Acc: 0.9984
val Loss: 1.4341 Acc: 0.8179

Epoch 54/69
----------
train Loss: 0.0024 Acc: 0.9997
val Loss: 1.5026 Acc: 0.8148

Epoch 55/69
----------
train Loss: 0.0026 Acc: 0.9992
val Loss: 1.6046 Acc: 0.8173

Epoch 56/69
----------
train Loss: 0.0044 Acc: 0.9981
val Loss: 1.3364 Acc: 0.8251

Epoch 57/69
----------
train Loss: 0.0016 Acc: 0.9995
val Loss: 1.4984 Acc: 0.8220

Epoch 58/69
----------
train Loss: 0.0005 Acc: 1.0000
val Loss: 1.5610 Acc: 0.8153

Epoch 59/69
----------
train Loss: 0.0017 Acc: 0.9995
val Loss: 1.3520 Acc: 0.8364

Epoch 60/69
----------
train Loss: 0.0015 Acc: 0.9994
val Loss: 1.5953 Acc: 0.8034

Epoch 61/69
----------
train Loss: 0.0007 Acc: 1.0000
val Loss: 1.6250 Acc: 0.8142

Epoch 62/69
----------
train Loss: 0.0005 Acc: 1.0000
val Loss: 1.5463 Acc: 0.8117

Epoch 63/69
----------
train Loss: 0.0012 Acc: 0.9997
val Loss: 1.5688 Acc: 0.8199

Epoch 64/69
----------
train Loss: 0.0008 Acc: 0.9998
val Loss: 1.4750 Acc: 0.8225

Epoch 65/69
----------
train Loss: 0.0006 Acc: 0.9997
val Loss: 1.6174 Acc: 0.8086

Epoch 66/69
----------
train Loss: 0.0003 Acc: 1.0000
val Loss: 1.6037 Acc: 0.8132

Epoch 67/69
----------
train Loss: 0.0009 Acc: 0.9997
val Loss: 1.5039 Acc: 0.8282

Epoch 68/69
----------
train Loss: 0.0041 Acc: 0.9992
val Loss: 1.5608 Acc: 0.8199

Epoch 69/69
----------
train Loss: 0.0008 Acc: 0.9998
val Loss: 1.5780 Acc: 0.8158

Training complete in 63m 10s
Best val Acc: 0.836429

and this is the confusion matrix