I think it should be correct because it works well on GPU 0.
I’m feeling that this problem is relevant to this: https://github.com/pytorch/pytorch/issues/689 15
But I have no idea where the problem is.