Hi! I am facing a similar problem with Titan X GPUs with Pytorch 0.4. I am running it in a docker and allocate 12GB shared memory. On using nn.Dataparallel I get:
Runtime error: NCCL error 1, unhandled cuda error.
I tried the iommu disable option, and have the latest nccl2 library installed.
I tried conda and pip installs as well, but they give the same NCCL 1 error. Sometimes, the code deadlocks and the GPUs show 100% utilization.
p2p bandwidth latency test passes.
Any help would be appreciated.
Thanks