Ah, that’s a great insight, @sgugger - thank you!
I know you mentioned, this is based on your experience, but perhaps you could recommend a good document to read about this?
It should be only on different GPUs, right? i.e. different CPUs won’t have this difference
Batch size is identical.
Indeed, different number of GPUs would make such an impact - but it should be easy to restrict to, say, one for comparison purposes.
The intention is to try and replicate the CI environment, with purchasing an identical GPU, so that the test results shown locally would be also the same on CI.