Since CUDA calls are asynchronous, you are basically timing the kernel launch if you remove the cpu() call.
If you want to time CUDA calls, you should synchronize the timer before starting and stopping using:
torch.cuda.synchronize()
start = time.time()
# your calls
torch.cuda.synchronize()
stop = time.time()