[WIP] Enable reproducibility for distributed trainings
# What does this PR do?
This PR ensures reproducibility for distributed trainings by setting seed for worker in dataloader and setting environment variables for cuda.

This PR is motivated by [this issue](https://github.com/huggingface/transformers/issues/16549#). 

## Who can review?
@saattrupdan @sgugger I am looking forward to your feedback
_The documentation is not available anymore as the PR was closed or merged._<|||||>> Thanks for working on this, that's an important feature! So as to not introduce a breaking change, and for clarity of the API, I'd personally vouch for not adding the `enable_determinism` flag to the `set_seed` method.
> 
> From the title of the method I understand it should set the seed, and that's it. I don't think it should do anything else. However, the `enable_determinism_for_distributed_training` method likely needs the seed to be set in order to benefit from full determinism, so I'd even push to have the `set_seed` method called inside the `enable_determinism_for_distributed_training`, adding a `seed` argument to that last method.
> 
> What do you think?

I like this idea. I can implement it after we reach a conclusion on it, however, it is not clear to me how to implement it. Could you point me to which parts of the code I need to change/pay attention not to break anything if we decide to go for this idea? <|||||>@sgugger Thanks for the pointers and sorry for not being so clear. I would like to know in which places `enable_full_determinism` should be called. Currently, `set_seed` is called several places in the codebase. I don't think these calls will be replaced with `enable_full_determinism`. 

With the latest commits, I have already addressed your pointers. Now I am waiting your feedback for where to call `enable_full_determinism` in the codebase. It is not called any place in the codebase right now.<|||||>There can be an added flag in the `TrainingArguments` and we can call this function instead of `set_seed` in the `Trainer`. Otherwise it will be for the users to use this one instead of `set_seed` in their own scripts (you should make it accessible in the main init by the way!)<|||||>@sgugger I think I have addressed all your comments. Is there anything that needs to be done for this PR?<|||||>Is it normal that 3 tests fail suddenly after a commit in a docstring? I couldn't understand why tests are failing.<|||||>Those are just flaky, no link to your PR. Thanks again for all your work on this!<|||||>@sgugger @hasansalimkanmaz I had a question about this PR - why is it necessary to set `CUDA_LAUNCH_BLOCKING`? This disables asynchronous execution of CUDA programs, but the cuda/pytorch docs don't mention it necessary for deterministic training? I do use it to get the "true" stack trace when there are device-side asserts but was wondering what role it plays in deterministic training. Many thanks!
<|||||>@alexcoca It's required to make some CUDA algorithms deterministic if the CUDA version is older than 10.2. I suppose it could be replaced by a CUDA version check somehow, and only using it if it's an old version?<|||||>@saattrupdan I would go for this approach, because running the CUDA programs in asynchronous mode will definitely slow things down beyond belief. I implemented this PR myself without the `CUDA_LAUNCH_BLOCKING` setting and will report if I manage to preserve determinism.<|||||>I experimented with training a dialogue state tracking model on the SGD corpus starting from Google's v1.1 T5 (220M) paramaters. I allowed the model to train for roughly two epochs and evaluated task oriented performance every 2k steps (max train steps was 12k). 

Ran 4 experiments: 2 in which I set the seed, and an additional 2 where I do roughly the same as `ensure_determinism` except setting `CUDA_LAUNCH_BLOCKING`. I also set `CUBLAS_WORKSPACE_CONFIG=':4096:8'`. Each experiment was trained on 2 A100-80GB with `cuda/11.4 openmpi/4.1.1/gcc-9.4.0-epagguv`, `pytorch 1.10` and transformers `4.19.2`. You can see below that I was able to reproduce the metrics in all runs and with no major performance hits. I guess that convolution benchmarking and non-det ops are less relevant for T5. With `4.18.0` the performance was wreaking havoc on the same seed, sign that the data ordering was the culprit. 

![image](https://user-images.githubusercontent.com/30216068/169503692-e85d8e0d-a592-422f-97c9-e1f0d26f1d25.png)
![image](https://user-images.githubusercontent.com/30216068/169505839-580ddbe7-f6b5-4fa4-93e7-826d71fa359a.png)

I guess the moral of the story here  is that one could:

- Check CUDA version to avoid running in blocking mode when not necessary
- potentially allow the user to specify which `CUBLAS_WORKSPACE_CONFIG` as `:16:8` may impact performance (see [here](https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility))#16907 

@sgugger ?<|||||>Agreed for the first one. For the second one, we could avoid overriding an existing `CUBLAS_WORKSPACE_CONFIG` if it's already in the env? In all cases, it should be clearly stated in the doc of the flag that triggers the full reproducibility that it comes at a performance price.<|||||>Yes, I agree with the above! I'm at ACL next week but I'll try and open a small PR to address this the week after!
<|||||>Thanks, @alexcoca for noticing this and for your time.