First, thanks for your reply. Because single GPU training model, its state dict is different from Mutilply GPU training model. so it can’t load state dict from single GPU resume model on the multiply GPU training model.