Non-deterministic training issue on GPU: TF-BERT
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
TF-BERT 
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
SST-2
* [ ] my own task or dataset: (give details below)

## To reproduce

In spite of combining learnings from:
* [the "complete recipe" in NVIDIA's slides from gputechconf](https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9911-determinism-in-deep-learning.pdf) 
* [a recently suggested workaround](https://github.com/tensorflow/tensorflow/issues/38185#issuecomment-643014439) for non-determinism issues with crossentropy loss

... I am still arriving at the following [short, non-deterministic colab notebook example](https://colab.research.google.com/drive/1VSU8lYFD0E1HKZrIL1MvyIRwAktlSF_t?usp=sharing) to train BERT .

My results for the sum of model weights (as computed with [this suggested function](https://github.com/NVIDIA/tensorflow-determinism/issues/2#issuecomment-548210203)) after training **for only 5 steps** is (differences are **`highlighted`** below):


| | Device | Before training | After training |
| ------------- | ------------- | ------------- | ------------- |
| Run 1  | GPU | -641227.5609667897224  | -641237.442 **`5159916282`** |
| Run 2  | GPU | -641227.5609667897224  | -641237.442 **`3093758523`** |
| | |  | |
| Run 1 | CPU | -641227.5609667301178 | -641238.1506845243275 |
| Run 2 | CPU | -641227.5609667301178 | -641238.1506845243275 |

This variance gets increasingly more pronounced when the model is trained for longer periods of time. 
I am expecting a general problem with the computational graph with BERT introducing non-determinism. As a result, this could affect a large part of the huggingface community. 

Please keep in mind that determinism is of key importance in certain industries and also a pre-requisite for reproducible research.

Could you please help identify the source of non-determinism and provide guidance on how we can resolve it?

Steps to reproduce the behavior:

1. Execute colab notebook above on the GPU runtime using Tensorflow 2.2.0, observe non-deterministic behavior
2. Execute colab notebook above on the CPU runtime using Tensorflow 2.2.0, observe deterministic behavior

## Expected behavior
Training should be deterministic both on GPU and CPU runtime for TF 2.2.0.

## Environment info
* tensorflow==2.2.0
* nlp==0.2.1
     
- `transformers` version: 2.11.0
- Platform: Linux, Ubuntu 18.04.3 LTS bionic
- Python version: 3.6.9
- PyTorch version (GPU?): -
- Tensorflow version (GPU?): 2.2.0-gpu
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Hi! [This StackOverflow issue](https://datascience.stackexchange.com/questions/14812/making-keras-tensorflow-code-execution-deterministic-on-a-gpu) might be of interest to you.

Namely:

> In fact, the randomness(non-determinstic) is a behavior of GPU.
> 
> The reason behind is that cuDNN(and othere CUDA stuffs) uses a non-deterministic algorithm to compute gradients, thus we can't determine anything.<|||||>Note that this issue is being addressed as an [issue in the tensorflow-determinism repo](https://github.com/NVIDIA/tensorflow-determinism/issues/19). I have also added a reference to that repo in the above-mentioned Stack Exchange / Data Science question.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
