The push to the CPU and back to GPU shouldn’t be a problem, as the id of the parameters shouldn’t change, thus the optimizer still holds valid references to the parameters.
However, I would suggest to use the same device after storing the state_dict, since internal states of the optimizer (e.g. using Adam) will also be stored on the initial device.