PyTorch 0.4 vs PyTorch 1.0

this should be expected.

>= 2 times on PyTorch 1.0.

This is weird, since you set both random seeds and deterministic cuda behavior. I am currently using the same cards (RTX 2080TIs) and couldn’t find a difference when running code multiple times. Sorry, I have no way to explain this. But one possibility could be that you set the random seeds somewhere later in your script after you initialized the weights?