I am trying to make my training code as deterministic and reproducible as possible. When running the same training code multiple times, and always re-initialising the model, I get different results - even if I set the seeds manually, before all runs start. I found that when I reset the seed on every training run, all runs do end up with the same result. This seems to indicate that torch (or numpy or Python’s random) internally does change its seed.
def set_seed():
    torch.manual_seed(3)
    torch.cuda.manual_seed_all(3)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(3)
    random.seed(3)
    os.environ['PYTHONHASHSEED'] = str(3)

for i in range(10):
    set_seed()
    model = init_model()
    model.train()
    model.test()

The code above always produces the same result for the test set, as expected. But when I only set the seed once, i.e. outside the loop, results vary over the iterations. What causes this?