Thanks for the answer.
Although I’m not sure if this could explain the problematic behavior.

The code runs on a single GPU in both cases. (the multi-gpu code didn’t train properly so I tried single-gpu training on both machines and got different results, and the test-code posted here is a simple single-gpu script)
In the full training script I checked the other, more complex, model and it trained the same in both machines. Only one model showed difference during training and it’s the model detailed in the test code.
The differences in the results I get are not insignificant (like the differences in float precision) so I think it’s not the cause either.

Note: It seems that the random input in my test code changes with different pytorch versions. The results here are for pytorch 1.1.0
The output of the multi-GPU with pytorch 1.2.0 (bad training):

$ python Test_different_training.py
Results of the forward pass on the first batch is same on both machines:
Same input:  tensor([[0.0807, 0.0398, 0.8724],
[0.3084, 0.7438, 0.3201],
[0.8189, 0.6380, 0.3528],
[0.9787, 0.5305, 0.4797],
[0.9665, 0.9392, 0.7120]], device=‘cuda:0’)
Same output tensor([[-0.3300,  0.0380],
[-0.0558,  0.1243],
[-0.0914,  0.0408],
[-0.1176,  0.0539],
[-0.1128,  0.2583]], device=‘cuda:0’, grad_fn=)
Results of the forward pass after 10 batches is diffrent:
Same input:  tensor([[0.8217, 0.3160, 0.9787],
[0.7174, 0.7287, 0.3107],
[0.2931, 0.7804, 0.8234],
[0.5223, 0.7749, 0.5192],
[0.2176, 0.2366, 0.8337]], device=‘cuda:0’)
Different output tensor([[-0.1803, -0.1522],
[-0.1187,  0.0910],
[ 0.0831, -0.0918],
[-0.0440,  0.0358],
[ 0.1193, -0.3293]], device=‘cuda:0’, grad_fn=)