Reproducibility issue with Trainer
Hi,
I am trying to fine-tune a model using PyTorch backend and Trainer class, but when I train 2 times with exactly the same parameters, I still get different results. I have supplied the seed and also used PyTorch-specific commands:
```py
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False
```

However, after all of these failed I went to Trainer's source code and noticed these lines (https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L566)
```
if self.args.world_size <= 1 and _is_torch_generator_available:
    generator = torch.Generator()
    generator.manual_seed(int(torch.empty((), dtype=torch.int64).random_().item()))
```
Is there any reason to actually sample the seed here? Wouldn't it hurt reproducibility between the runs? I'm not sure if this is a bug or not, which is why I didn't open this issue as a bug.
Also do you have more suggestions on where to look for sources of randomness using PyTorch backend?

Thank you very much in advance!

Best regards,
Dmytro
Hi,

Googling "reproducibility Huggingface trainer" returns this helpful post on our forum: https://discuss.huggingface.co/t/fixing-the-random-seed-in-the-trainer-does-not-produce-the-same-results-across-runs/3442<|||||>Hi Niels!

Thanks for your swift response! I've seen the post, but it seemed people still couldn't make it work judging from the last message in the thread. I've tested setting the seed before instantiating a model now and fixed a couple of other bugs of my own and now I managed to get reproducible results.

However, I still can't grasp why the original Trainer sets the seed to a random value in the code snippet provided in the original post of this thread, since this generator is used to instantiate a sampler, which is passed to a DataLoader, in turn. Won't that hurt reproducibility?

Best,
Dmytro<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.