If your state_dict contains CPU tensors, you’ll trigger the host to device copy twice: first for the model parameters, then again for the state_dict.
If that’s the case, you could first restore the model and push it only once to the GPU.
How often are you resetting the model that you worry about the performance?
Usually you’ll setup the model once and the training will take much longer than the initialization.