Hi,
We have a machine with 4 Titan X GPUs - but we can only train Pytorch models on the 3 GPUs other than the primary one. As long as we utilize the primary GPU, pytorch would hangs after a few iterations and nvidia-smi would report GPU loss afterwards - only rebooting can recover the machine.
We have tried to uninstall X-org from Ubuntu 16 desktop, or re-install Ubuntu 16 server without X. Disconnect display etc. It always causes GPU loss if the training utilizes the primary GPU.
We also tried to use MXNET train on 4 GPUs, it goes well without seeing this problem.
Any idea why Pytorch cannot train on the primary GPU? Any workaround we could try?
Thanks!
Ming