As CUDA calls are asynchronously, you would have to synchronize before starting and stopping the timer:
torch.cuda.synchronize()
t0 = time.time()
y = net(x)
torch.cuda.synchronize()
t1 = time.time()

Could you add it and try it again?