python, numpy and pytorch RNG states to the same states as they were at the moment of saving that checkpoint,
which should make the "stop and resume" style of training as close as possible to non-stop training.
However, due to various default non-deterministic pytorch settings this might not fully work. If you want full
determinism please refer to Controlling sources of randomness. As explained in the document, that some of those settings
that make things deterministic (.e.g., torch.backends.cudnn.deterministic) may slow things down, therefore this
can't be done by default, but you can enable those yourself if needed.
Trainer Integrations
The [Trainer] has been extended to support libraries that may dramatically improve your training
time and fit much bigger models.
Currently it supports third party solutions, DeepSpeed and FairScale, which implement parts of the paper ZeRO: Memory Optimizations
