Could you describe the issue a bit more or post a dummy code snippet, which explains the issue?
Based on your sentence it seems that something like this:
# 1.
model = MyModel()
model.load_state_dict(torch.load(...))
# 2.
optimizer = torch.optim.SGD(model.parameters())
optimizer.load_state_dict(torch.load(...))
# 3. 
model.eval()
output = model(input)

works better when #2 is skipped?