Hi folks,
I’ve noticed that fixing the seed in the Trainer does not produce the same results across multiple training runs.
For example, suppose we fix the seed in the TrainingArguments and instantiate a model and trainer as follows:
batch_size = 16
model_checkpoint = "distilbert-base-uncased"
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)

args = TrainingArguments(
    "test-glue",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=3,
    seed=123
)

trainer = Trainer(
    model,
    args,
    train_dataset=train_ds,
    eval_dataset=valid_ds,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

If I fine-tune this on COLA, I get the following results
Epoch	Training Loss	Validation Loss	Matthews Correlation	Runtime	Samples Per Second
1	0.517900	0.467562	0.455635	0.740500	1408.496000
2	0.335300	0.500026	0.490934	0.686200	1519.946000
3	0.232300	0.618692	0.493626	0.693100	1504.833000

Now suppose we re-instantiate the model and trainer, and fine-tune:
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)
trainer = Trainer(
    model,
    args,
    train_dataset=train_ds,
    eval_dataset=valid_ds,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
trainer.train()

I would expect to get exactly the same results, but instead find small differences in the outputs:
Epoch	Training Loss	Validation Loss	Matthews Correlation	Runtime	Samples Per Second
1	0.519000	0.473168	0.458883	0.702900	1483.911000
2	0.335800	0.502644	0.486363	0.701400	1487.070000
3	0.229200	0.616023	0.497130	0.710800	1467.258000

My best guess is that the DataLoader is the source of the difference since it uses a RandomSampler. However, I would have thought that fixing the seed would also fix that.
Although these differences are tiny, having reproducible runs really helps stay sane during debugging and I’m wondering whether anyone here knows how to fix this?
For context, I’m working in Jupyter notebooks and here is a small Colab notebook from which I produced the above numbers: https://colab.research.google.com/drive/15nv40o81JfKwubFBVOjZkqk8bRauXig8?usp=sharing 7