Do I understand correctly from your answer " My evaluation script is pretty standard. with torch.no_grad(): model.eval() …" that you compare the results if the two behave exactly the same at evaluation time?

Yes.

I am afraid floating point numbers are not associative and doing things in a different order will lead to difference in the result.

Good to know. Very interesting, since these where completely distinct calculations. I think as you say, my training is not that stable, if it leads to this gap.
Thank you for your help!