Solved by ptrblck in post #2 
              
              
                Since CUDA operations are performed asynchronously, your time measurements are most likely a bit off. 
You should call torch.cuda.synchronize() before starting and stopping your timer.