All “slow” lines contain a cpu() call, which will create a synchronization if your script runs on the GPU.
To properly time CUDA code, you should synchronize before starting and stopping the timer (if you are manually profiling).
torch.cuda.synchronize()
t0 = time.time()
...
torch.cuda.synchronize()
t1 = time.time()

You could also use the profiler 3 to measure the execution of your code.