One more question, during the execution, did you set resume to True or False? I am not confident if ddp_model.load_state_dict can restore all DDP states properly. We don’t have tests covering that yet. It might be safer to save ddp_model.module and then reconstruct DDP instances from the loaded ddp_model.module.