Okay, I made a new conda environment and installed the nightly build with conda install pytorch torchvision cudatoolkit=9.2 -c pytorch-nightly as suggested by https://pytorch.org/get-started/locally/ 1.
I re-run the script, and I get similar result to before insofar as the outputs do not match:
tensor(-0.0064, device='cuda:1') tensor(-0.0003, device='cuda:1') tensor(0.0064, device='cuda:1')
tensor(-0.0040, device='cuda:0') tensor(0.0005, device='cuda:0') tensor(0.0056, device='cuda:0')
tensor(-0.0064, device='cuda:0') tensor(-0.0003, device='cuda:0') tensor(0.0064, device='cuda:0')
tensor(3.0040, device='cuda:1')
tensor(3.0049, device='cuda:1')

And again, if I re-run the last two lines again manually, I get a different output:
tensor(3.0058, device='cuda:1')
tensor(3.0067, device='cuda:1')

In this case though, both of these output values change with every run, rather than just the first one.
@ptrblck I have never done these kinds of checks before so I don’t know if it was running correctly before. This was the first time I remember that I’ve tried loading a model trained on one GPU onto another.
If both of your set-ups get the correct results, and mine still doesn’t even with the nightly build, what else could it be? I haven’t explicitly set cudnn to deterministic, but I assume that neither of you did either? There must be some mistake I’m making somewhere.