mohotmoz:

My general understanding is that NVSwitch allows uniform memory access to all GPUs from all GPUs, so you can think of it as “one giant GPU” with a continuous memory space.


No, NVSwitch uses a more sophisticated connectivity between the devices as described here 1. It doesn’t “automatically” create a single device.



 mohotmoz:

I have tried to recompile pytorch replacing CUDAMalloc with CUDAMallocManaged as in this paper
[…]
But PyTorch still pins the memory to one device (and if it grows beyond that, it spills into CPU memory not onto the other GPUs).


That would be expected, as Unified memory 4 is used.