Solved by JoelSjogren in post #2 
              
              
                Duplicate of https://github.com/pytorch/pytorch/issues/15793. 

You are not measuring timing accurately. CUDA launches are asynchronous… 


torch.cuda.synchronize()