Well, it seems that when I do not load optimizer.state_dict() the training loss still continue from the last checkpoint.
And I am using Adam, how can this happened?
This is the log without loading optimizer state:
[Epoch 0/200] [Batch 156/6000] [D loss: 0.766469] [G loss: 1.088398, pixel: 0.005941, adv: 0.494318] ETA: 17:56:19.482757

Here is with loading optimizer state:
[Epoch 0/200] [Batch 187/6000] [D loss: 0.767510] [G loss: 0.981094, pixel: 0.005185, adv: 0.462558] ETA: 18:16:18.970563