I want to resume training and try to load the state_dict of an Adam optimizer, however I find that the loss will increase for a few hundred iterations before going down again.
Later I found that the self.state in torch.optim.Adam is always empty which causes state initialization for each optimizer.step() call 7. Shall we modify the Adam code by adding self.state[p] = state in the end of the loop?
I’m using ignite for training and its ModelCheckpoint 2 for saving model’s and optimizer’s state_dict(). I found all the saved optimizers’ self.state is an empty dict().