How are you profiling the code?
Make sure to synchronize the code before starting and stopping the timer using toch.cuda.synchronize().
Since CUDA operations are asynchronous, the cpu() call will create a synchronization point, so that you might be in fact profiling some other workload which is still executed on the GPU.