Thanks for the answer, I am curious does loading optimizer.state_dict() will make huge difference? Because I found no different between loading and not loading optimizer.state_dict in both fine-tuning and ‘continue’ training.