Loading from checkpoint without skipping requires the same number of GPUs
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.5
- Platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.9.0a0+2ecb2c7 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

When changing the number of GPUs available, the trainer fails to load from the checkpoint that was trained with a different number of GPUs checkpoint. 
Note that this does not happen when the flag ignore_data_skip is used.

The "per_device_train_batch_size" remains unchanged between runs.

I use Trainer API 


The error looks like this:

 File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1298, in train
    self._load_rng_state(resume_from_checkpoint)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1529, in _load_rng_state
    torch.cuda.random.set_rng_state_all(checkpoint_rng_state["cuda"])
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/random.py", line 73, in set_rng_state_all
    set_rng_state(state, i)
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/random.py", line 64, in set_rng_state
    _lazy_call(cb)
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py", line 114, in _lazy_call
    callable()
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/random.py", line 61, in cb
    default_generator = torch.cuda.default_generators[idx]
IndexError: tuple index out of range


Thank you


Yes, in general the API to resume training requires you to have the exact same setup. You won't get to the same results otherwise.<|||||>> Yes, in general the API to resume training requires you to have the exact same setup. You won't get to the same results otherwise.

My aim is not to reproduce the results. 
From time to time I have to switch between different hardware setups and I wish to continue from a checkpoint. Is it at all possible or I cannot work around this error?<|||||>This would require some digging into the Trainer to fix the error you get and see if there are others that arise. It's fixable but not high priority on our side so it might take a little bit of time.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale, will whip something around soon to avoid the error.<|||||>> 

Thank you!