After 2nd iteration: always same result when training in a loop
I train a BERT model on a binary classification task. I do the training 4 times in a row. With same train and validation data and with the exact same hyperparameters. I use the default seed in TrainerArguments and set no other seeds.

The results of the 2nd, 3rd and 4th iteration are 100% the same. The result of the 1st run is unique. This behavior is 100% reproducible.

It is not clear why this is the case. Since I set a seet and work with same data.

PIP Libs I use (no conda libs):
- sentencepiece-0.1.91 
- tokenizers-0.9.3 
- transformers-3.5.1
- torch-1.7.0+cu101

Screenshot:
![grafik](https://user-images.githubusercontent.com/229382/99275325-a57bf080-282b-11eb-843d-fb8c38dc8e3d.png)

Colab to reproduce:
https://colab.research.google.com/drive/1HjQ7p5AlDY9kcWo7uSXzteh7x1ustEdN?usp=sharing

I already did some facy stuff after each iteration:
```python
    del trainer
    del training_args
    del model
    del config
    del train_result
    del tokenizer
    del labeled_dataset_test
    del labeled_dataset_train
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()  
```

But that does not help. Can someone please help me here and claify whats up?

PS: I do not think that a seed can be the reason. If a seed would be the reason the 1st and 2nd run would also be the same and AFAIK a GPU training is not 100% deterministic. So a small difference would still remain.

Here the same code as a .py script with transformers 3.4.0 (instead of 3.5.1), CUDA 11.0 and torch 1.7.0

![grafik](https://user-images.githubusercontent.com/229382/99278875-e249e680-282f-11eb-8ecd-b6d3c691bc38.png)
<|||||>Same problem also happening with torch 1.6.0+cu101 - see here: https://colab.research.google.com/drive/1-9jcAsFGf79kpiCSQa4QaBdXvZQstE_n?usp=sharing

![grafik](https://user-images.githubusercontent.com/229382/99285590-04dffd80-2838-11eb-9956-98528e279fd8.png)
<|||||>Same Bug also with torch==1.5.1+cu101 and transformers==3.3.1

see here: https://colab.research.google.com/drive/1HqMOQ_UzGI4z_OWOd0qFsfdVpKfLbHaM?usp=sharing<|||||>Ok I think I found the reason why this happens.

When `TrainingArguments` is called with default params a seed is not just used to init the network but also set everywhere else.
This is done by calling this: https://github.com/huggingface/transformers/blob/c89bdfbe720bc8f41c7dc6db5473a2cb0955f224/src/transformers/trainer_utils.py#L29

After that point everything that should be randomized in the next iteration like shuffling the training data and so on is not random anymore but dependent on the seed. Since this seed is set again and again to the same value everything seems to be deterministic and not random anymore. The reson why the 1st iteration has a different value is because the seed is set relativly late after data is loaded.

Closing this...
<|||||>Well - I was thinking about this:

There is a default param "hidden" in a class like `TrainingArguments`. This default value sets all seeds of all libs like python default, numpy, torch to a fixed value. This can be the source of some nasty problems because it removes randomness where nobody would have ever suspected it.

Best positive example are the `sklearn.model_selection` classes (functions). Most of them accept a seed but they just use the seed internaly and do not set it in every module you could think of.

I am not sure if I would call this a bug but at least it is an issue. That's why I reopen.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
