Since CUDA calls are asynchronous, you might be just measuring the time to launch kernels.
Although I’m currently not sure if the to() op is asynchronously run be default.
However, add a synchronization before starting and stopping the timer:
torch.cuda.synchronize()
t0 = time.time()
...
torch.cuda.synchronize()
t1 = time.time()

If you are using a DataLoader, you might set pin_memory=True to use pinned memory, which will speed up the transfer between CPU and the GPU. Here 5 is a good explanation of what’s going on under the hood.