Do you load the best checkpoint and try to continue the training?
If so, what optimizer are you using and how are you handling it?
Are you recreating the optimizer or saving/loading like your model?
Some optimizers have internal states (e.g. per-parameter lr, etc.), which will be reinitialized when recreated, which might explain the issues.