Reproducible checkpoint
# What does this PR do?

This PR fixes reproducibility when resuming training from a checkpoint for PyTorch >= 1.6 and non-TPU trainings. The current implementation in the Trainer had two flaws:

First when going over the data at the beginning (skipping epochs and batches), the first shuffling was the same in the original training or resuming from the checkpoint, but not the subsequents shufflings. This is because those shufflings were determined by the global torch RNG which had a different state at the end of epoch 1 in a full training vs just after the shuffle of epoch 1 in the resumed training. I did not identify the exact reason but there was one or several calls to the global torch RNG during epoch 1 of the full training and when resuming we only called for the shuffle of epoch 1.

To fix this, a generator is now used to determine the shuffling, this way the generator is set with the same seed in both cases (full training or resuming from checkpoint) and then has the exact same number of calls (one shuffle per epoch) so that generator ends up in the same state in the full training or when resuming from a checkpoint.

The second thing that was different is the CUDAs RNG states, which are used in the dropout layer of the models. To fix this, all RNG states are saved when saving a checkpoint and set after all the data skip phase when resuming training.

Fixes #11504
Fixes #11323
@stas00 Could you double check the last changes are all good with you? They include loading/saving all RNGs state (and separately in distributed training) as you suggested.<|||||>Everything looks great, @sgugger.

Given the fragile nature of this added feature it'd be relatively easily to break it, so may I suggest that at least some rudimentary test would go a long way to prevent this?

(fragile, not because how it was coded, but because it's just fragile in its own random nature)

perhaps duplicating test_trainer's resume test and generating a random number on each np, pt, py before and after resume? <|||||>Added a new doc section in a separate PR https://github.com/huggingface/transformers/pull/11588

