I have been observing that transformers quite often generates different qualitative outcomes on different hardware, despite using the same seed and the same configuration.
Shouldn’t it be the case that identical code with a fixed seed and identical configuration (batch_size, etc.) should produce identical results no matter the hardware?
For example here, @sshleifer  gets a bleu score of 27.65 on his PR branch, whereas I get 27.84. The only difference is hardware.
Another example, we have been battling at finding hparams that will make CI happy with pl_glue_run.py test - I was getting acc/f1 of 1.0 on my hardware but CI was getting 0.5, despite multiple attempts to improve it. Two attempts were made (1, 2) but the test is currently still testing acc>0.25, so really we are just testing that the example runs.
I have seen a lot of others examples, these are just 2 recent ones I could easily point to.
Perhaps some of you have practical insights at how this can be improved.
Thank you.