I have a deterministic code that gives the same result between runs on the same machine.
I tried to launch it on two machines:

my macbook pro m1 with intel python 3.8 and torch=1.9.0
google colab with python 3.7 and torch=1.9.0+cu102

It gives different results in most cases.
However, if I set the optimizer to Adam and use 10 epochs at most, the results are the same.
Even weirder, if I use CUDA on colab (which needs torch.use_deterministic_algorithms(False) but the code still gives the same results every time), then I can do 109 epochs at most that give the same results between the two machines.
If I use SGD, then using CPU on colab works for 9 epochs at most and GPU doesn’t work when any epoch is done.
I couldn’t make a MRO, here is the code 1. It is hard to dive down because I make most operations using a package. But anyone can probably try to run it and report their results that are different. Also, when relaunching I cannot get the outputs to match as I explain above.
The notebook gives rrmse: 1.179134459111933 and my computer gives rrmse: 1.1791344531807701.
Does anyone have an idea?