Until now I didn’t bother to restore the previous state of my Adam Optimizer to continue training.
In fact, I observed that by doing this, a new best performing model is often found after this “hard” continuation.
Is there anything I am missing that makes this practice absolutely wrong?