@albanD
My GPUs are connected with NVLINK 2.0 150GB/s to the CPUs. This is to your statement “the PCIe connection is the limit, and our current transfer speed is pretty close to it (not sure exactly how close).”
So I assume that CUDA Unified Memory in Pytorch on my system architecture could have a slightly better benefit compared with the one you described.
Rgds,
FM