I just tried to reproduce this issue and it seems to work fine:
model = models.resnet50(pretrained=True)

data = torch.randn(10, 3, 224, 224)
target = torch.randint(0, 1000, (10,))

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

# Train for 5 epochs
for epoch in range(5):
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print('Epoch {}, loss {}'.format(epoch, loss.item()))

# Predict on validation data
val_data = torch.randn(10, 3, 224, 224)
model.eval()
pred1 = model(val_data)

# Save and restore model
torch.save(model.state_dict(), 'tmp.pt')

model = models.resnet50(pretrained=False)
model.load_state_dict(torch.load('tmp.pt'))

# Predict on same validation data
model.eval()
pred2 = model(val_data)

# Compare
print((pred1 == pred2).all())
> tensor(1, dtype=torch.uint8)

Could you compare your code with mine and check for differences or if I’m missing something?
How large is the difference? Note that you might encounter some floating point precision issues.