Solved by cwitkowitz in post #5 
              
              
                Thanks for pointing me in the right direction. You were right about the IO_PAGE_FAULT messages being shown in dmesg. I was able to fix the problem by following ngimel’s comment in Multi-GPU K80s · Issue #1637 · pytorch/pytorch · GitHub. 
In my previous comment, I was wondering whether NCCL is requir…