Pushing a CUDATensor to the CPU is a synchronizing operation in the default setup and will thus accumulate the time unless you manually synchronize the code.
You could try to use the non_blocking option, but would still need to wait until the data transfer is done and based on your code you are directly using the CPUTensor afterwards, so you wouldn’t gain any perf. improvements.