How are you testing?
Moving to CPU is going to potentially require synchronization as most GPU operations are asynchronous. So it depends on whether the GPU result is actually there and the time will often be due to other operations that aren’t yet completed.
It’s also likely to depend a fair bit on bus latency which will depend on connection method and also other load on the system. Are they both NVLink or both PCIe? Performance will likely differ a lot between the two.
In general though you want to avoid waiting on CPU transfers. Not actually accessing the CPU tensor immediately I think generally results in not synchronising when you call .cpu() (though would love clarification here), or you can explicitly do an asynchronous transfer to CPU (a search will find details, but think the basics are to pin your destination host memory and use dest.copy_(src, non-blocking=True). Though I think this was only enabled in PyTorch 1.2 due to a previous bug 6. That issue should also have or link to full code.