Solved by ptrblck in post #2 
              
              
                Pushing a CUDATensor to the CPU is a synchronizing operation in the default setup and will thus accumulate the time unless you manually synchronize the code. 
You could try to use the non_blocking option, but would still need to wait until the data transfer is done and based on your code you are dir…