If you are timing GPU operations, note that CUDA calls are asynchronous, so that you have to add synchronization points before starting and stopping the timer:
torch.cuda.synchronize()
t0 = time.time()
# your operations
torch.cuda.synchronize()
t1 = time.time()

Most likely you are measuring the CUDA initialization time etc. in GPU_1.