I am using PyTorch v1.0.0 and seem to be running into the same issue as described in the original post. Duplicate processes are started on GPU 0 from processes running on the other GPUs. See screenshot for an example.

Screen Shot 2018-12-15 at 9.20.03 PM.jpg966×1486 882 KB