Could you try to run the code without any available GPUs via export CUDA_VISIBLE_DEVICES= before running the script?
I assume the load method is smart enough to copy the device state_dict back to your CPU model.
EDIT: This issue doesn’t seem to be reproducible using the latest nightly binary, so you might just update.