Note that CUDA operations are called asynchronously, so that you should add manual synchronization points before starting and stopping the timer using torch.cuda.synchronize().
Otherwise you might just time the kernel launch.