I ran into an issue when using 16 K80 GPUs today. It turns out that the default behavior (peer-to-peer) in PyTorch will not support more than 8 devices, probably for very good reasons that I don’t understand. A proposed solution is to set NCCL_P2P_LEVEL=1 for the environment, but I’m not sure how to actually do that because I have never had to fiddle with NVIDA environment. Is there a PyTorch command that will let me set NCCL_P2P_LEVEL? Where do I change this? I’m building models in a Jupyter Notebook on a Windows machine.