Cannot Achieve Reproducability with Tensorflow Transformer Models
I've been experimenting with the roberta-large model with both PyTorch and TensorFlow for a sentiment analysis task. With the PyTorch model, I am able to achieve 100% reproducability, however, this is not the case with the Tensorflow model.

I have set all the necessary seeds as follows:

```
seed_val = 3
os.environ['PYTHONHASHSEED'] = str(seed_val)
np.random.seed(seed_val)
random.seed(seed_val) 
tf.random.set_seed(seed_val)
```
and I am even using the fix described at https://github.com/NVIDIA/tensorflow-determinism:

```os.environ['TF_DETERMINISTIC_OPS'] = '1'```

I am not sure if reproducability is fully supported with the TensorFlow transformer models yet, or if I am doing something wrong.

Here is the link to the **CoLab notebook** containing my code:
https://drive.google.com/open?id=1xPTYPl8LyRrMgkiXUNtSbxxJ7pAslD2x

Thanks in advance
I have the same issue and I get (sometimes wildly) different results from run to run. Here's my code:

https://github.com/dmitriydligach/Thyme/blob/master/RelKeras/et.py

Does anybody have a solution yet?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
