Annus_Zulfiqar:

And one more thing, I have only changed the classifier at the end and didn’t use bn there. The rest of the model is the same standard implementation that comes with pytorch models module


I can’t give exact solution.
But here is my few suggestions
1.Check each class level accuracy.Gives you for different test batches what are things the model not doing well.
https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html 57
class_correct = list(0. for i in range(10))
class_total = list(0. for i in range(10))
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs, 1)
        c = (predicted == labels).squeeze()
        for i in range(4):
            label = labels[i]
            class_correct[label] += c[i].item()
            class_total[label] += 1


for i in range(10):
    print('Accuracy of %5s : %2d %%' % (
        classes[i], 100 * class_correct[i] / class_total[i]))

2.If you dont shuffle you will get bad numbers.Because you see all the labels in your test and train sets.Sometimes the labels will be in order of the classes. which will lead your model to see same class until sometime(Eg:1-100) and then some other class(Eg: 100-200) which will lead to catastrophic forgetting.It’s always better to send as shuffled batches