Solved by ptrblck in post #2 
              
              
                CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping the code manually via torch.cuda.synchronize() or use a built-in utility such as torch.utils.benchmark from the nightly/master, which will add it for you (as well as warmup iterations)…