Note as well that the .to() functions are the only blocking operations on the CUDA api that is completely asynchronous otherwise.
So if you try to time your code without using torch.cuda.synchronize() properly, it is expected that the .to() functions are the only ones that will take a significant amount of time ! But this is only because these functions are waiting on the GPU to finish processing all other operations before returning.