Load optimizer state on CPU to avoid CUDA OOM
# What does this PR do?

As reported in #22123 resuming from a checkpoint can get a user out of memory if the optimizer state is loaded directly on GPU. This PR loads it on CPU by default and it will be copied over to the proper device by PyTorch in `load_state_dict`. This might be a bit slower at the checkpoint loading time (so just once) but will benefit users training large models.

Fixes #22123 
_The documentation is not available anymore as the PR was closed or merged._