I guess this link will provide you relative answer

  
    
    
    Measuring GPU tensor operation speed 
  
  
    Hi, 
I would like to illustrate the speed of tensor operations on GPU for a course. 
The following piece of code: 
x = torch.cuda.FloatTensor(10000, 500).normal_()
w = torch.cuda.FloatTensor(200, 500).normal_()

a = time.time()
y = x.mm(w.t())
b = time.time()
print('batch GPU {:.02e}s'.format(b - a))

a = time.time()
y = x.mm(w.t())
b = time.time()
print('batch GPU {:.02e}s'.format(b - a))

prints 
batch GPU 1.06e-01s
batch GPU 3.43e-04s

so I presume that there is some “lazy operations” delayed…