@bhushans23 Thank you for the reply, I figured out a solution. I got the problem was due to difference in architechture in training and inference codes. So in the inference code I changed,
model = resnet18(pretrained = True)

to
model = resnet18(num_classes = 4)

This solved the problem!