Thanks for pointing me in the right direction. You were right about the IO_PAGE_FAULT messages being shown in dmesg. I was able to fix the problem by following ngimel’s comment in Multi-GPU K80s · Issue #1637 · pytorch/pytorch · GitHub 1.
In my previous comment, I was wondering whether NCCL is required in general (not only for the tests), since everything seemed to be working without it (besides my issue in this post). Could you please elaborate on how/when NCCL would be beneficial?