Of course the big O constant is small – memory copies are fast.
But the big O constant is still significant. Memory is a bottleneck – CPUs spend most of their time waiting for registers to be filled, not waiting for computation to be finished.
Try to minimize the number of CPU <=> GPU transfers. I believe you want to use async=True in cuda or cpu when you can.