Yes, I use model.eval().
I solved the issue. The problem was that the model was trained using DataParallel and when I was trying to load the state dictionary there was a mismatch between the dictionary keys of the model and the network even though I set the strict=False.
Once I remove the .module from the state_dict keys it solved the issue.